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Executive  Summary 


In  the  current  en  route  Air  Traffic  Control  (ATC)  system,  the  ATC  Specialist  (ATCS)  has 
primary  responsibility  for  safe  and  efficient  traffic  flow.  The  expansion  of  the  National  Route 
Program  (Mtf )  will  allow  airlines  more  flexibility  in  filing  and  amending  flight  plans.  The 
increased  flexibility  for  the  airlines  will  likely  move  the  ATCS  away  from  direct  control  to  a 
managerial  position.  Programs  like  the  NRP  may  make  the  ATCS  a  monitor  that  ensures  that 
aircraft  adequately  separate  themselves.  Researchers  and  ATCSs  have  voiced  concern  about  the 
change  from  active  control  to  a  more  monitoring  role.  These  concerns  include  a  reduction  in 
situation  awareness  (S  A),  memory,  and  vigilance. 

This  experiment  placed  ATCSs  at  two  levels  of  involvement.  At  one  end,  they  controlled  traffic 
as  they  normally  would  in  the  field.  At  the  other  level,  they  monitored  traffic,  but  did  not 
actively  control  or  communicate  with  aircraft.  For  both  levels  of  involvement,  we  conducted 
simulations  with  moderate  and  high  traffic  load. 

The  simulated  sector  was  generic.  It  was  easy  to  learn,  but  it  still  enabled  the  experimenters  to 
create  complex  scenarios.  The  generic  airspace  had  the  advantage  that  ATCSs  from  anywhere 
within  the  continental  United  States  could  participate.  Using  ATCSs  from  several  Air  Route 
Traffic  Control  Centers  may  make  results  more  applicable.  The  study  investigated  the  effect  of 
the  change  in  involvement  and  task  load  by  measuring  eye  movements,  workload,  S  A,  system 
performance,  ATCS  performance  ratings,  organization  of  information  in  memory,  and  responses 
to  questionnaires. 

The  results  of  this  study  are  varied.  The  changes  in  involvement  and  task  load  did  not  affect  eye 
movement  characteristics,  although  they  did  influence  the  structure  in  the  visual  scanning 
pattern.  Measures  that  capture  eye  movement  characteristics  (e.g.,  the  number  and  duration  of 
blinks  and  fixations)  did  not  change.  The  probability  that  a  controller  fixates  objects  in  a 
particular  order  is  an  indication  of  the  structure  of  the  visual  scan.  Using  these  transition 
probabilities,  we  found  that  the  structure  in  the  visual  scan  changed  as  a  function  of  involvement 
and  load.  The  experiment  may  have  been  too  short  to  alter  well-rehearsed  scanning  behavior  to 
change  eye  movement  characteristics.  It  is  clear  that  the  ATCSs  looked  longer  per  glance  at 
aircraft  than  any  other  object. 

Measured  workload  correlated  well  with  traffic  volume,  bi  addition,  workload  actually 
decreased  when  ATCSs  monitored  instead  of  actively  controlled  traffic.  They  had  less  to  do,  and 
the  measures  reflected  this.  In  this  study,  ATCSs  received  a  relief  briefing  as  they  would  in  the 
field.  Analysis  of  the  data  revealed  that  workload  was  lower  during  the  first  5  minutes  than 
subsequent  5-minute  intervals. 

The  ATCS  S  A,  as  measured  by  the  response  time  to  SA-related  questions,  was  lower  under 
monitoring  conditions  than  under  active  control.  Under  active  control,  the  level  of  traffic  load 
did  not  influence  S A,  but,  under  monitoring  conditions,  a  higher  traffic  load  led  to  a  sharp 
decrease  in  S  A.  This  is  a  critical  finding  with  potential  implications  for  future  training. 

Two  observers  rated  the  ATCSs  performance  under  active  control  conditions.  The  observers 
indicated  that  the  ATCSs  provided  adequate  ATC  information  under  both  levels  of  traffic  load. 
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The  observers  felt  that  the  quality  of  prioritization  suffered  from  the  increase  in  traffic  load.  In 
the  observers’  opinion,  the  ATCS  SA  was  lower  under  high  traffic  load.  A  change  in  load  did 
not  affect  the  quality  of  conununications  nor  did  it  affect  the  safe  and  efficient  flow  of  traffic. 
Interestingly,  the  observers  found  that  an  increase  in  traffic  load  reduced  the  ATCS  exhibited 
knowledge  of  the  letters  of  agreement  and  standard  operating  procedures.  It  is  likely  that,  under 
the  increased  pressure  of  a  higher  traffic  load,  the  ATCSs  were  less  capable  in  applying  their 
knowledge. 

To  assess  how  ATCSs  organize  information  in  memory,  we  asked  the  ATCSs  to  place  data 
blocks  back  to  the  position  that  represented  the  last  screen  update  of  the  simulation.  The  current 
study  did  not  reveed  changes  in  memory  organization  across  levels  of  involvement  and  traffic 
load.  However,  the  percentage  of  data  block  positions  correctly  recalled  under  active  control 
was  higher  than  under  monitoring. 

After  each  simulation,  the  controllers  filled  out  a  questionnaire.  Their  responses  indicated  that 
they  perceived  active  control  scenarios  to  be  more  difficult  and  more  realistic.  The  ATCSs 
perceived  that  their  SA  did  not  suffer  from  the  change  in  involvement.  They  indicated  that  their 
S A  for  aircraft  positions  and  potential  violations  was  not  as  good  under  high  traffic  load 
conditions  for  both  active  control  and  monitoring  conditions. 

The  expected  changes  in  programs  like  the  NRP  may  move  the  ATCS  to  a  situation  that  will  fall 
somewhere  between  the  current,  active  control  situation  and  the  simulated,  monitoring  situation 
of  this  study.  The  results  indicated  that,  although  perceived  workload  was  less  under  monitoring 
conditions,  the  objective  SA  measures  showed  that  ATCS  S A  declines  substantially.  The  fact 
that  the  ATCS  may  not  have  been  aware  of  the  reduction  in  S  A  suggests  that  a  monitoring 
situation  without  SA  enhancers  is  not  a  good  idea. 

Although  our  experiment  was  too  brief  to  alter  eye  movement  characteristics,  the  visual  scanning 
patterns  showed  changes.  These  small  changes,  after  only  a  brief  exposure  to  work  as  a  monitor, 
may  suggest  changes  in  eye  movement  characteristics  while  monitoring  for  longer  periods. 
Changes  in  visual  scanning  are  an  indication  of  visual  information  retrieval  strategies.  The 
altered  SA,  in  combination  with  a  change  in  information  retrieval  strategies,  warrants  careful 
examination  of  the  need  for  training  and  assistance  for  situations  where  the  ATCS  is  no  longer  in 
active  control. 
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1.  Introduction 


The  current  air  traffic  control  (ATC)  system  will  undergo  significant  changes  in  equipment  and 
procedures  in  the  near  future  (Federal  Aviation  Administration  (FAA),  1996, 1997).  The 
proposed  changes  will  affect  the  role  of  current  ATC  Specialists  (ATCSs).  One  of  the  significant 
proposed  changes  is  the  implementation  of  the  expansion  of  the  National  Route  Program  (NRP). 
Some  ATCSs  refer  to  the  NRP  expansion  as  Free  Flight  (Smith  et  al.,  1998).  Within  the  FAA, 
Free  Flight  is  now  an  accepted  term  where  airlines  and  pilots  obtain  more  freedom  in  amending 
flight  plans.  In  Free  Flight,  the  ATCSs  function  may  involve  more  monitoring  with  less  direct 
control.  Ground-  and  satellite-based  navigational  aids  such  as  the  Traffic  Alert  and  Collision 
Avoidance  System  (TCAS),  global  positioning  systems  (GPS),  the  Wide  Area  Augmentation 
System  (WAAS),  and  Automatic  Dependent  Surveillance-Broadcast  (ADS-B)  will  make  Free 
Flight  possible.  Implementation  will  involve  three  levels  that  include  free  scheduling,  free 
routing,  and  free  maneuvering  (FAA,  1996).  The  implementation  of  Free  Hight  as  described  by 
RTCA  (1995)  is  significant  because  the  ATCS  will  do  more  monitoring  as  opposed  to  active 
control  under  the  current  system.  Each  subsequent  stage  may  reduce  the  active  role  of  the 
ATCS.  The  transition  from  active  control  to  monitoring  could  have  a  significant  impact  on 
ATCS  behavior  and  performance  in  general. 

1.1  Background 

The  primary  responsibility  for  the  separation  of  aircraft  in  the  current  en  route  ATC  system 
belongs  to  the  ATCS.  A  number  of  tools  help  the  ATCS  maintain  separation  between  aircraft 
including  the  plan  view  display  (PVD)  and  the  flight  progress  strip  (FPS).  These  tools  assist  the 
ATCS  in  developing  and  maintaining  an  understanding  of  the  current  and  future  air  traffic 
situation.  Using  specific  knowledge  of  the  current  situation  and  general  knowledge  of  ATC,  the 
ATCS  manages  air  traffic  within  a  sector.  In  the  current  ATC  system,  the  ATCS  plays  an  active 
role.  Pilots  must  follow  all  ATCS  instructions  and  assigned  flight  plans.  Pilots  can  make 
deviations  (e;g.,  changes  in  heading,  altitude,  and  route)  only  with  the  approval  of  the  ATCS  or 
in  an  emergency. 

The  implementation  of  Free  Flight  as  conceived  years  ago  moved  the  ATCS  from  an  active 
participant  in  the  separation  of  aircraft  to  a  monitor  that  ensures  that  aircraft  adequately  separate 
themselves.  In  Free  Flight,  the  ATCS  may  become  an  air  traffic  manager.  Therefore,  it  was 
important  to  examine  how  such  a  transition  will  affect  ATCS  behavior  and  performance. 

Hopkin  (1988)  has  argued  that  active  participation  in  memory  and  understanding  is  more 
important  than  researchers  thought  in  the  past.  He  suggested  that  it  is  necessary  to  preserve  the 
interaction  between  an  operator  and  the  task  at  hand.  One  way  to  do  this  is  to  require  the 
performance  of  an  additional  task  while  monitoring  to  compensate  for  the  lack  of  active 
involvement.  The  additional  task  would  keep  the  operator  “in  the  loop.”  It.  should  serve  to  help 
maintain  relevant  knowledge  about  the  current  situation. 

A  number  of  studies  support  Hopkin’s  (1988)  interaction  hypothesis.  For  example.  Held  and 
Freedman  (1963)  and  Slamecka  and  Graf  (1978)  demonstrated  that  memory  is  better  for 
something  that  you  do  yourself  than  for  something  done  for  you.  Conversely,  a  series  of  studies 
challenged  Hopkin’s  interaction  hypothesis  (Albright,  Truitt,  Barile,  Vortac,  &  Manning,  1994; 
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Vortac,  Edwards,  Fuller,  &  Manning,  1994;  Vortac,  Edwards,  Jones,  Manning,  &  Rotter,  1993). 
These  studies  focused  primarily  on  the  impact  of  removing  flight  progress  strips  rather  than  on 
Free  Flight.  However,  they  supported  the  view  that  the  reduction  of  workload  can  improve  or 
maintain  cognitive  functioning  despite  an  associated  reduction  of  active  interaction  with  the  task 
at  hand. 

Vigilance  is  another  concern  when  reducing  the  amount  of  active  ATCS  involvement.  When 
operators  perform  a  task  for  any  length  of  time,  especially  when  monitoring  a  situation,  it  is 
difficult  to  sustain  an  optimal  level  of  focused  attention  (Parasuraman,  1986).  The  vigilance 
decrement  is  the  inability  to  remain  focused.  Many  simulated  and  operational  radar/sonar 
monitoring  studies  have  provided  evidence  for  a  vigilance  decrement  (Baker,  1962;  Colquhoun, 
1977;  Schmidtke,  1976;  Thackray,  Bailey,  &  Touchstone,  1979;  Thackray  &  Touchstone,  1989). 
However,  a  vigilance  decrement  usually  occurred  only  after  a  considerable  amount  of  time  (e.g., 
2  hours).  Other  research  has  shown  the  evidence  for  the  occurrence  of  a  vigilance  decrement 
after  only  a  short  period  (Stem,  Boyer,  Schroeder,  Touchstone,  &  Stoliarov,  1994). 

Additionally,  vigilance  decrements  may  vary  as  a  function  of  load  (Stem  et  al.;  Thackray  et  al.). 
Regardless  of  the  results  of  past  vigilance  research  in  monitoring  behavior,  the  ATC  system  has  a 
responsibility  for  public  safety.  Researchers  should  not  ignore  issues  like  vigilance  decrement  as 
a  possible  consequence  of  Free  Flight  implementation. 

Free  Flight  could  diminish  the  amount  of  active  involvement  of  the  ATCS.  Diminished  active 
involvement  may  affect  cognitive  processing  of  information  and  vigilance.  Because  of  these 
concerns,  one  must  consider  how  to  assess  the  potential  impact  of  the  original  concepts  of  free 
flight. 

1.2  Objective 

This  study  assessed  the  impact  of  a  change  in  load  and  level  of  involvement  on  the  behavior  and 
performance  of  ATCSs.  We  assessed  if  ATCSs  can  maintain  their  awareness  (or  picture)  when 
their  level  of  direct  involvement  declines.  In  addition  to  participant  questionnaires  and  objective 
and  subjective  performance  measures,  we  examined  ATCS  behavior  and  cognitive  processing 
through  the  assessment  of  eye  movements,  situation  awareness  (SA),  and  memory. 

In  the  current  study.  Full  Performance  Level  (FPL)  en  route  ATCSs  operated  in  a  simulated, 
generic,  en  route  environment.  Simulation  offers  complete  situational  control  and  measurement 
during  a  simulated  traffic  scenario. 

2.  Method 


For  convenience,  we  have  presented  the  appendixes  in  the  following  manner:  A-C  (Airspace- 
related),  D-F  (Questionnaires),  G  (Participant  Instructions),  H-N  (Detailed  result  tables),  and  O 
and  P  (Coordination  events  and  situation  presence  assessment  method  (SPAM)  queries). 

2.1  Participants 

Sixteen  FPL  ATCSs  from  12  Air  Route  Traffic  Control  Centers  (ARTCCs)  within  the  United 
States  served  as  voluntary  participants.  All  participants  were  non-supervisory,  full-time  FPL 
ATCSs.  None  of  the  participants  was  on  medical  waiver  or  in  a  staff  position  at  the  time  of  the 
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experiment.  Eleven  participants  had  normal  vision  and  five  had  corrected-to-normal  vision.  The 
mean  age  of  the  participants  was  37.3  years  (29-53).  They  were  FPLs  for  a  mean  of  9.3  years 
(2.5-17)  and  had  worked  in  their  current  facility  for  10.9  mean  years  (6-22).  Six  participants  had 
worked  at  more  than  one  facility  during  their  ATC  career.  The  participants  worked  air  traffic  for 
an  average  of  1 1.7  months  out  of  the  previous  12  months.  Using  a  10-point  scale,  the 
participants  rated  their  current  skill  level  as  8.2  (6-10),  stress  level  during  the  past  several  months 
as  5.6  (3-9),  motivation  to  participate  in  the  study  as  8.9  (6-10),  and  their  current  state  of  health 
as  8.8  (5-10). 

2.2  Experimental  Staff 

Three  human  factors  specialists  (HFSs)  conducted  the  study.  One  of  the  HFSs  started  and  ended 
the  simulations,  conducted  the  SPAM  measurements,  and  issued  between  sector  coordination 
requests.  The  second  HFS  provided  ATCSs  with  the  Post-Scenario  Questionnaires  (PSQs), 
instracted  ATCSs  on  how  to  use  instruments,  and  started  the  Recall  procedure.  The  third  HFS 
performed  the  eye  movement  measurements  and  data  analysis. 

Two  subject  matter  experts  (SMEs)  participated  in  the  study.  Both  SMEs  were  active 
supervisory  controllers  in  ARTCCs.  During  the  simulations,  the  two  SMEs  conducted  an  over- 
the-shoulder  (OTS)  evaluation  of  controller  performance  and  recorded  the  correct  answers  to 
question  asked  as  part  of  the  SPAM. 

Three  simulation  pilots  entered  commands  into  the  simulator  and  read  back  clearances  in 
response  to  ATCS  instructions.  Engineering  support  staff  at  the  FAA  William  J.  Hughes 
Technical  Center  Research  Development  and  Human  Factors  Laboratory  (RDHFL)  monitored 
the  simulations  and  ensured  proper  function  of  equipment  and  software. 

2.3  Materials 

2.3.1  Airspace 

The  airspace  used  for  the  experiment  was  Genera  Center  High  (Guttman,  Stein,  &  Gromelski, 
1995).  ^nera  Center  High  (Figure  1),  hereafter  referred  to  as  Genera  sector,  is  a  synthetic 
airspace  sector  developed  to  be  representative  of  a  high  altitude,  en  route  sector.  Genera  sector 
and  its  related  elements  are  easy  to  learn  while  still  allowing  for  considerable  complexity. 
Jetways,  fixes,  intersections,  and  airports  have  simple  names  for  ease  of  memorization. 

Appendix  A  contains  the  Genera  sector  Standard  Operating  Procedures  (SOPs),  and  Letter  of 
Agreement  (LOA)  with  Charlie  Center.  Appendix  B  contains  a  description  of  Genera  sector 
airspace. 

2.3.2  Scenarios 


Each  participant  controlled  four  practice  and  four  experimental  scenarios.  The  complexity  of  the 
scenario  and  rate  at  which  aircraft  entered  the  airspace  constituted  load.  The  development  of 
scenarios  occurred  in  close  consultation  with  an  SME  to  ensure  the  desired  levels  of  complexity 
and  realism.  Each  scenario  began  with  traffic  in  the  airspace  similar  to  that  present  after  a 
position  relief  briefing. 
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GENERA  SECTOR:  ADJACENT  SECTORS  AND  FACILITIES 


Figure  1.  Genera  Center  high  sector  map. 

The  four  practice  scenarios  had  a  moderate  level  of  load.  These  scenarios  allowed  the 
participants  to  become  familiar  with  the  airspace  and  equipment  used  during  the  experiment. 
During  practice  scenarios,  aircraft  entered  the  airspace  at  the  rate  of  about  1.5  every  2  minutes. 
Each  practice  scenario  lasted  40  minutes.  Four  coordination  events  (Appendix  O)  occurred 
during  each  practice  scenario.  We  simulated  coordination  events  by  ringing  the  landline.  When 
the  ATCS  answered  the  landline,  one  of  the  experimenters  responded  with  a  between-facility 
coordination  message  (e.g.,  “This  is  Tech  Center,  requesting  higher  for  USA6255”). 

The  four  experimental  scenarios  consisted  of  two  active  control  and  two  monitoring  scenarios. 
The  two  Active  Control  (A)  scenarios  simulated  air  traffic  and  procedures  similar  to  a  current 
field  setting.  One  of  the  scenarios  was  High  Load  (HL)  and  one  was  Low  Load  (LL).  During 
the  HL  scenario,  aircraft  entered  the  airspace  at  an  average  rate  of  one  per  minute.  The  LL 
scenario  consisted  of  aircraft  entering  the  airspace  at  an  average  rate  of  about  one  every  2 
minutes.  Each  A  scenario  contained  three  coordination  events. 

The  two  Monitor  Control  (M)  scenarios  approximated  conditions  similar  to  an  advanced  stage 
(free  maneuvering)  of  Free  Flight.  One  scenario  was  HL  and  the  other  was  LL.  Load  varied  for 
M  scenarios  in  the  same  manner  as  for  A  scenarios.  During  M  scenarios,  aircraft  traversed  the 
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airspace  without  assistance  from  the  ATCS.  Aircraft  had  flight  plans  and  navigated  through  the 
airspace  to  avoid  conflicts  with  other  aircraft.  Data  block  updates  and  handoffs  took  place 
automatically.  M  scenarios  also  contained  three  coordination  events. 

2.4  Equipment 

2.4.1  Hardware 

2.4. 1 . 1  Oculometer  and  Headtracker 

An  Applied  Science  Laboratories  (ASL)  series  4000  oculometer  recorded  eye  movements.  The 
ASL  4000  oculometer  compensates  for  head  movement  by  using  a  magnetic  tracker  (The  Bird™, 
Ascension  Technologies  Corporation). 

2.4. 1 .2  Console  Configuration 

The  experiment  used  a  single  en  route  ATC  workstation.  A  2,000  by  2,000  pixel,  29”  video 
display  unit  presented  the  radarscope  PVD.  A  19”  monitor  mounted  above  the  PVD  displayed  a 
map  of  the  airspace.  An  Air  Traffic  Workload  Input  Technique  (ATWIT)  device  (Stein,  1985) 
mounted  to  the  immediate  left  of  the  PVD  within  easy  reach  of  the  participant  allowed  input  of 
workload  ratings.  The  workstation  had  a  full  flight  strip  bay  to  the  right  of  the  PVD,  an  en  route 
keyboard,  and  a  trackball  with  three  buttons.  A  landline  allowed  interfacility  and  intrafacility 
communications.  A  software  program  implemented  an  electronic  version  of  the  Quick  Action 
Keys  (QAKs)  and  Computer  Readout  Device  (CRD)  in  the  upper  right  hand  comer  of  the  PVD. 
To  activate  a  QAK,  the  participant  had  to  move  the  cursor  to  the  appropriate  QAK  and  depress 
the  left  button  on  the  trackball.  The  center  button  on  the  trackball  allowed  the  participant  to 
make  entries  on  (or  slew  on)  an  aircraft.  The  right  trackball  button  served  as  a  home  key  that 
would  return  the  cursor  to  the  center  of  the  PVD  when  pressed. 

2.4. 1.3  Communications  Configuration 

The  communication  system  linked  us  with  the  ATCS,  SMEs,  and  simulation  pilots.  We  could 
communicate  with  the  simulation  pilots  and  SMEs  without  distracting  the  participant.  The 
participants  made  transmissions  by  depressing  a  handheld  thumb  switch. 

2.4.2  Software 


ATCoach  (UFA,  Inc.,  1992)  software  simulated  the  air  traffic  scenarios.  ATCoach  is  a  high 
fidelity,  dynamic  ATC  simulator  that  enabled  a  realistic  design  and  control  of  airspace  and 
scenarios. 

The  Data  Reduction  and  Analysis  (DRA)  program  reduced  simulation  data  provided  by 
ATCoach  and  integrated  the  data  with  information  about  the  airspace  and  communication-events 
data.  The  output  of  the  DRA  for  en  route  simulations  contained  detailed  information  on 
conflicts,  complexity,  errors,  communications,  and  load.  The  DRA  provided  summary  data  for 
the  entire  simulation  or  specific  intervals. 
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2.5  Design  and  Procedure 

2.5.1  Experimental  Design 

The  main  experimental  design  employed  a  2  X  2  (load  X  involvement)  within-subjects  design. 
The  ATCSs  worked  the  practice  and  experimental  scenarios  in  a  counterbalanced  order.  Each 
participant  worked  one  of  eight  condition  orders  for  both  the  practice  and  experimental 
scenarios.  We  counterbalanced  the  four  practice  scenarios  for  presentation  order.  The  ATCSs 
worked  the  four  experimental  scenarios  in  an  order  counterbalanced  for  condition  only. 

Experimental  scenarios  required  either  A  or  M  control  by  the  participant.  Experimental 
scenarios  were  of  either  HL  or  LL.  The  questions  that  relate  to  changes  in  performance  and 
behavior  are  as  follow: 

a.  Does  scanning  behavior  differ  across  experimental  conditions? 

b.  Do  subjective  workload  ratings  (ATWIT)  differ  across  experimental  conditions? 

c.  Do  SME  ratings  differ  across  scenarios? 

d.  Do  responses  to  PSQs  differ  across  scenarios? 

e.  Do  performance  scores  differ  across  task  load  levels? 

2.5.2  Dependent  Measures 

To  evaluate  the  effect  of  changes  in  load  and  level  of  involvement  on  ATCS  performance  and 
behavior,  we  collected  data  on  eye  movements,  workload,  S  A,  and  performance.  The  following 
subsections  introduce  the  variables  collected  in  the  respective  data  sets. 

2.5.2. 1  Eve  Movements 

The  variables  calculated  to  characterize  eye  movements  were  fixations,  saccades,  blinks,  and 
pupil  size.  To  characterize  the  visual-scanning  pattern,  we  calculated  measures  of  conditional 
information  or  structure.  Section  3.1  provides  a  detailed  description  of  the  background,  results, 
and  discussion  of  the  eye  movement  measurements  used  in  this  study. 

2.5.2.2  Air  Traffic  Workload  Input  Technique 

An  ATWrr  device  (Stein,  1985)  recorded  response  latencies  (e.g.,  times  to  respond)  and 
workload  ratings  during  all  conditions.  The  participant  made  a  rating  on  the  ATWIT  device 
every  5  minutes.  Before  each  rating,  a  tone  alerted  the  participant  who  then  had  20  seconds  to 
make  a  workload  rating.  The  participants  used  a  scale  of  1  (low  workload)  to  10  (high 
workload).  ATWIT  is  a  reliable  and  relatively  unobtrusive  on-line  measure  of  subjective 
workload.  Section  3.2  provides  a  detailed  description  of  the  background,  results,  and  discussion 
of  ATWrr  ratings  and  response  latencies. 
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2.5.2.3  Situation  Present  Assessment  Method 

The  experiment  used  the  SPAM  (Durso  et  al.,  1995).  We  presented  six  queries  (Appendix  P) 
during  each  experimental  scenario  at  a  rate  averaging  one  every  5  minutes.  We  presented  the 
queries  using  a  simulated  landline.  During  each  scenario,  half  of  the  queries  related  to 
conceptual  information  regarding  the  present  situation,  and  half  of  the  queries  related  to 
conceptual  information  regarding  future  information.  For  example,  a  present  query  was,  “Which 
aircraft  currently  has  a  higher  altitude,  USA335  or  TWA790?”  A  future  query  was,  “Which 
aircraft  will  reach  the  MEDLE  intersection  first,  SWG321  or  AAL123?”  No  two  queries  asked 
about  the  same  aircraft.  Each  particular  scenario  dictated  the  order  of  present  versus  future  query 
types.  We  developed  the  queries  in  consultation  with  an  SME  and  based  them  on  information 
considered  relevant  and  meaningful  to  the  participant.  We  recorded  the  time  it  took  the 
participant  to  answer  the  landline,  then  read  them  the  query,  and  recorded  their  answers.  The 
SMEs  independently  scored  each  response  as  correct  or  incorrect.  Section  3.3  provides  a 
detailed  description  of  the  background,  results,  and  discussion  of  the  SPAM. 

2.5. 2.4  Real  Time  Objective  Performance 

The  experimental  conditions  included  several  objective  and  subjective  performance  measures 
referred  to  as  the  Real  Time  Objective  Performance  (RTOP).  The  RTOP  provided  a  means  to 
assess  ATCS  skill  and  strategy.  These  measures  were  meaningful  only  in  the  A  condition.  The 
data  reduction  module  can  break  down  performance  data  by  conflicts,  complexity,  error, 
communications,  and  load.  Analyses  involved  only  a  subset  of  these  performance  variables. 
Section  3.4  provides  a  detailed  description  of  the  background,  results,  and  discussion  of  the 
performance  measures. 

2.5.2.5  Subject  Matter  Expert  Ratings  Forms 

All  experimental  scenarios  involved  both  subjective  and  objective  SME  ratings.  Two  SMEs 
made  ratings  independently  on  ratings  forms  (Appendix  C).  The  SMEs  provided  on-line 
performance  ratings  using  the  rating  forms  developed  by  Sollenberger,  Stein,  and  Gromelski 
(1996).  They  derived  the  OTS  items  from  the  standard,  on-the-job-training,  evaluation  form 
(FAA  Form  3120-25)  normally  used  during  training.  Section  3.5  provides  a  detailed  description 
of  the  background,  analysis,  and  discussion  of  the  rating  forms. 

2.5.2.6  Recall 

After  each  experimental  scenario,  the  participants  recalled  the  contents  of  the  airspace  as  it 
existed  when  the  scenario  ended.  They  were  to  associate  data  blocks  with  the  respective  beacon 
returns  as  quickly  and  accurately  as  possible.  The  exercise  involved  all  data  blocks  associated 
with  aircraft  that  were  in  the  airspace  or  otherwise  under  their  control.  They  were  to  guess  if 
they  were  not  certain  about  a  response. 

Using  the  same  display  that  served  as  the  PVD,  the  participants  saw  a  representation  of  the 
airspace  including  beacon  returns,  vector  lines,  and  leader  lines  for  each  aircraft  located  in  their 
exact  position  as  when  the  scenario  ended.  A  bin  at  the  bottom  of  the  display  contained  the  data 
blocks  involved  in  the  exercise  in  random  order.  Using  the  trackball,  the  participant  selected  a 
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data  block  from  the  bin.  They  placed  the  data  block  with  the  beacon  return  to  which  they 
believed  it  belonged.  Dark  gray  squares  indicated  areas  in  which  to  place  data  blocks.  They 
used  the  left  trackball  button  to  select  and  place  the  data  blocks.  The  participants  used  as  much 
time  as  needed  to  complete  the  task.  Software  recorded  selection,  placement  times,  and  response 
accuracy  for  each  data  block.  Section  3.6  provides  a  detailed  description  of  the  background, 
results,  and  discussion  of  the  Recall  measures. 

2.5.2.7  Questionnaires 

The  experiment  included  questionnaires  to  solicit  demographic  information  and  opinions  from 
the  participants.  We  used  three  self-administered  questionnaires  adapted  from  Willems,  Allen, 
and  Stein  (in  press). 

a.  The  Entry  Questionnaire  (Appendix  D)  collected  information  about  the  participants.  It 
included  items  relating  to  ATCS  experience,  skill,  stress,  motivation,  and  health. 

b.  The  PSQ  (Appendix  E)  solicited  information  from  the  participants  about  each  particular 
scenario.  It  included  items  relating  to  realism,  difficulty,  and  performance  for  a  particular 
scenario.  Section  3.7  provides  a  detailed  description  of  the  background,  results,  and 
discussion  of  the  PSQ  measures. 

c.  The  Post-Experimental  Questionnaire  (PEQ)  (Appendix  F)  obtained  general  opinions 
from  each  participant  regarding  the  experiment  as  a  whole. 

2.5.3  Procedure 

2.5.3. 1  Weekly  Schedule  of  Events 

Experimenters  collected  data  from  two  participants  each  week.  ATCS  #1  arrived  on  Tuesday 
morning  and  finished  the  last  simulation  on  Wednesday  morning.  ATCS  #2  went  through  the 
same  schedule,  abut  started  Wednesday  afternoon  and  finished  Thursday  afternoon.  Table  1 
depicts  the  schedule  for  this  collection  procedure. 

2.5.3 .2  Training 

Training  consisted  of  classroom  and  practical  hands-on  training.  The  participant  and  the 
experimental  staff  were  present  for  the  training  sessions. 

With  an  SME,  we  conducted  the  classroom  instruction.  First,  we  obtained  verbal  consent  and 
then  informed  the  ATCS  of  the  right  to  withdraw  from  the  experiment  at  any  time.  The 
participant  then  completed  the  entry  questionnaire,  and  we  provided  initial  information  about  the 
schedule  of  events.  We  showed  the  participant  the  oculometer  to  be  worn  during  all  scenarios 
and  instructed  that  we  would  record  all  activities  on  videotape.  The  SME  briefed  the  participant 
on  the  equipment  used  during  the  study  (i.e.,  ATCoach,  the  Soft  Computer  Readout  Device 
(SCRD),  trackball,  and  landline)  and  the  Genera  sector,  SOPs,  and  LOAs. 
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Table  1.  Weekly  Schedule  of  Events 


Tuesda] 

Wednesday 

Thursday 

Time 

Event 

Time 

Event 

Time 

Event 

8:30 

Welcome  &  Entry  0  ATCS#1 

8:00 

Sim  Review 

8:15 

Sim  Review 

9:00 

Sector  &  Equipment  Briefing 

8:15 

Exp.  scenario  2 

8:30 

Practice  scenario  4 

10:00 

Bie^ 

9:15 

Break 

9:15 

Break 

10:15 

Practice  scenario  1 

9:30 

Exp.  scenario  3 

9:30 

Exp.  scenario  1 

11:00 

Break 

10:30 

Break 

10:30 

Break 

11:15 

Practice  scenario  2 

10:45 

Exp.  scenario  4 

10:45 

Exp.  scenario  2 

12:00 

Lunch 

11:45 

Exit  0  &  Debrief 

11:45 

Lunch 

13:30 

Practice  scenario  3 

12:30 

Lunch 

13:00 

Exp.  scenario  3 

14:15 

Break 

13:00 

Welcome  &  Entry  0  ATCS#2 

14:00 

Break 

14:30 

Practice  scenario  4 

13:30 

Sector  &  Equipment  Briefing 

14:15 

Exp.  scenario  4 

15:15 

Break 

14:30 

Break 

15:15 

Break 

15:30 

Exp.  scenario  1 

14:45 

Practice  scenario  1 

15:30 

Exit  0  &  Debrief 

16:30 

Data  backup 

15:30 

Break 

16:00 

Data  backup 

15:45 

Practice  scenario  2 

16:30 

Break 

16:45 

Practice  scenario  3 

17:30 

Data  backup 

After  the  classroom  instruction,  the  participant  engaged  in  hands-on  training.  A  very  simple  air 
traffic  scenario  (five  aircraft)  started.  The  participant  then  activated  all  of  the  functions  of  the 
SCRD  and  displays.  These  functions  included  the  flight  plan  readout,  route  readout,  J-ring,  data 
block  updates  (temporary  and  assigned  altitude  updates,  vector-line  length  changes,  leader-line 
length  changes),  and  data  block  handoff  and  acceptance.  We  demonstrated  how  the  landline 
worked.  Once  the  participant  understood  how  to  use  all  of  the  workstation  functions,  we 
explained  the  function  of  the  oculometer. 

Each  participant  engaged  in  four  40-minute  practice  scenarios.  We  gave  instructions  pertaining 
to  the  ATWTT  device  (Appendix  G).  The  participant  wore  the  oculometer  during  all  practice 
scenarios  to  acclimate  to  its  presence.  Two  SMEs  independently  completed  the  rating  forms 
during  all  practice  scenarios.  After  each  scenario,  we  removed  the  oculometer,  and  the 
participant  completed  a  PSQ.  To  give  the  participant  some  experience  in  using  the  human- 
computer  interface,  we  introduced  the  recall  procedure  at  the  end  of  the  fourth  practice  scenario. 

2.5.3.3  Data  Collection 

Experimental  data  collection  began  after  completion  of  the  fourth  practice  scenario.  Before  each 
experimental  scenario,  the  participant  received  instructions  about  the  specific  condition  (A  or 
M),  the  ATWrr  device,  the  SPAM,  and  the  recall  procedure. 
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Before  A  conditions,  the  participant  received  instructions  to  control  traffic  as  normally  in  the 
field.  Before  M  scenarios,  the  participant  received  instructions  to  simply  monitor  the  traffic. 
During  M  conditions,  the  participant  could  perform  all  functions  that  were  normally  available. 
The  instructions  for  the  M  conditions  were  intentionally  vague  to  see  how  the  participant  would 
behave  during  monitoring. 

Researchers  informed  the  participants  that  the  ATWIT  device  emits  a  brief  tone  every  5  minutes. 
When  the  tone  sounded,  the  participant  had  20  seconds  to  press  a  number  on  the  touch-sensitive 
screen  indicating  the  current  level  of  workload.  A  selection  of  1  would  indicate  the  lowest  level 
of  workload,  and  a  10  would  indicate  the  highest  level  of  workload.  If  the  participant  did  not 
make  a  selection  within  20  seconds  of  the  alerting  tone,  the  software  automatically  assigned  a 
rating  of  10. 

The  participant  was  also  aware  that  SPAM  used  only  one  landline  during  the  scenarios  and  that 
this  landline  served  all  coordination  purposes  between  the  participant  and  adjacent  sectors  or 
centers.  At  various  times,  a  call  came  over  the  landline  from  the  ‘Tech  Center.”  An  intermittent 
tone  over  a  loudspeaker  next  to  the  ATCS  workstation  indicated  an  incoming  landline  call.  Once 
the  participant  answered  the  landline  by  pressing  a  key  on  the  communications  panel,  we  asked  a 
SPAM  forced  choice  question.  The  participant  had  to  answer  the  query  as  quickly  and 
accurately  as  possible. 

The  presentation  of  the  query  did  not  interrupt  the  scenario,  and  the  participant  could  use  all 
available  information  to  answer  the  question.  Each  scenario  included  six  queries  that  occurred  at 
approximately  5-minute  intervals.  In  addition  to  the  six  queries,  we  made  three  other  landline 
calls  that  required  a  coordination  of  activity  from  the  participant.  Coordination  and  queries 
intermingled  to  prevent  the  participzmt  from  expecting  a  query  each  time  there  was  an  incoming 
landline  call.  Finally,  the  participant  received  instructions  about  the  recall  procedure  at  the  end 
of  the  scenario. 

After  we  gave  all  instructions  and  answered  any  questions,  calibration  of  the  oculometer  began. 
We  placed  the  oculometer  on  the  participant’s  head,  and  a  calibration  screen  consisting  of  17 
numbered  dots  appeared  on  the  monitor.  Following  our  instruction,  the  participant  had  to  hold  as 
still  as  possible  while  looking  at  each  dot  in  turn.  We  then  tested  the  qudity  of  the  calibration  by 
having  the  participant  look  at  a  subset  of  the  17  dots.  If  the  calibration  was  poor,  we  recalibrated 
the  oculometer.  If  the  calibration  was  acceptable,  the  experimental  scenario  began. 

We  began  each  experimental  scenario  with  a  short  count  down  over  the  communication  system. 
On  our  cue,  the  participant  touched  the  start  button  on  the  ATWIT  device,  and  the  simulation 
pilots  started  the  scenario.  An  SME  sitting  to  the  left  of  the  participant  gave  the  participant  a 
position  relief  briefing.  The  briefing  lasted  about  30  seconds  during  which  time  the  simulation 
pilots  did  not  make  any  calls  to  the  participant.  While  the  position  relief  briefing  took  place,  the 
second  SME  sitting  to  the  right  of  the  participant  near  the  FPS  bay  updated  the  FPS  markings. 
Once  the  briefing  was  complete,  the  participant  took  full  control  of  the  scenario.  Both  SMEs 
remained  in  the  room  in  order  to  complete  the  rating  form  and  score  the  SPAM  queries.  Each 
experimental  scenario  lasted  30  minutes. 
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The  recall  procedure  took  place  at  the  end  of  the  scenario.  The  participant  continued  to  wear  the 
oculometer  during  the  recall  procedure.  We  instructed  that  the  participant  would  see  a 
representation  of  the  airspace  as  it  appeared  when  the  scenario  ended.  All  radar  returns,  vector 
lines,  and  leader  lines  appeared  in  their  respective  and  proper  positions  as  when  the  scenario 
ended.  We  informed  the  participant  that  the  program  had  placed  data  blocks  for  all  aircraft  that 
were  in  the  airspace  or  otherwise  under  control  at  the  end  of  the  scenario  in  a  bin  at  the  bottom  of 
the  display.  The  participant  was  to  place  each  data  block  back  into  its  proper  position  as  quickly 
and  accurately  as  possible.  The  participant  could  use  as  much  time  as  needed  to  complete  the 
recall  procedure. 

After  the  participant  signaled  that  the  recall  was  complete,  we  removed  the  oculometer,  and  the 
participant  completed  Ae  PSQ.  The  next  scenario  began  after  a  break  of  approximately  15 
minutes.  We  continued  the  procedure  until  the  participant  completed  all  four  experimental 
scenarios. 

After  completion  of  the  experimental  scenarios,  we  all  returned  to  the  classroom  where  the 
participant  completed  the  PEQ.  We  then  debriefed  the  participant  by  further  explaining  the 
motivation  behind  the  experiment  and  answered  any  questions  about  the  experiment. 

3.  Data  Set  Specific  Analyses.  Results,  and  Discussions 

To  keep  the  background,  results,  and  discussion  related  to  a  specific  data  set  in  close  proximity 
of  one  another,  we  report  them  under  Subsections  3.1  through  3.7.  We  conducted  multivariate 
analyses  of  variance  (MANOVAs)  for  ATWIT  ratings,  performance  measures,  eye  movements, 
and  questionnaires.  We  tested  the  Wilks’  A  statistic  using  a  level  of  p  <  .05.  We  reported  the 
equivalent  F  statistic.  If  the  results  of  the  MANOVA  were  statistically  significant  [p  <  .05),  we 
performed  univariate  analyses  of  variance  (ANOVAs)  to  determine  which  of  the  dependent 
variables  were  significantly  different  across  experimental  conditions.  We  based  the  significance 
of  an  ANOVA  result  on  an  adjusted  alpha  level  using  the  following  formula: 

alpha(overall)  =  l-(l-alpha(individual))"  where  n  is  the  number  of  variables 

or: 

alpha(individual)  =  l-(l-alpha(overall))^^" 

We  reported  the  adjusted  alpha  level  with  each  analysis.  If  the  result  of  an  ANOVA  was 
statistically  significant,  we  performed  appropriate  post  hoc  tests  to  determine  which  conditions 
were  responsible  for  the  significance.  Figure  2  depicts  an  example  of  the  analysis  process. 

Other  researchers  have  used  a  more  lenient  approach  when  investigating  the  effects  of 
manipulation  on  dependent  variables  by  not  adjusting  the  alpha  level.  Such  an  approach  may 
inflate  the  overall  alpha  level  but  allows  researchers  to  investigate  trends  in  the  data.  In  the 
current  study,  we  follow  such  an  approach  to  investigate  trends. 
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Figure  2.  Statistical  analysis. 

3 . 1  Eve  Movements 

3.1.1  Background 

Researchers  have  used  eye  movements  previously  to  examine  behavior  within  the  context  of 
ATC.  Stein  (1992)  defined  visual  scanning  as  “a  systematic  and  continuous  effort  to  acquire  all 
necessary  visual  information  in  order  to  build  and  maintain  a  complete  awareness  of  activities 
and  situations  which  may  affect  the  ATCSs  area  of  responsibility”  (p.  3).  Researchers 
recognized  eye  movements  as  a  useful  measure  for  ATC  as  early  as  1975  (Karston,  Goldberg, 
Rood,  &  Sulzer,  1975).  Issues  of  complexity,  cost,  and  intrusiveness  have  resulted  in  few  ATC 
studies  using  eye  movements  as  a  dependent  measure  (Stein,  1989). 

Stein  (1992),  for  example,  compared  experience  (FPL  vs.  Developmental),  taskload  (LL  vs.  HL), 
and  oculometer  use  (Yes  vs.  No)  in  a  high  fidelity,  simulated  Terminal  Radar  Approach  Control 
(TRACON)  environment.  Results  showed  that  busier  ATCSs  had  shorter  and  more  frequent 
saccades,  and  FPLs  tended  to  make  more  fixations  than  did  Developmentals.  Compared  to 
controlling  air  traffic  without  the  oculometer,  wearing  the  oculometer  resulted  in  more  conflicts 
for  the  Developmentals  but  fewer  conflicts  for  the  FPLs. 

Stein  (1992)  used  three  measures  of  eye  movements  that  are  relatively  unique:  visual  efficiency, 
eye  motion  workload,  and  pupil  motion  workload.  Visual  efficiency  was  the  proportion  of 
scanning  time  spent  in  fixations.  Eye  motion  workload  was  the  average  degrees  each  second  that 
the  eyes  moved  during  the  course  of  each  scenario.  Pupil  motion  workload  was  the  cumulative 
difference  between  pupil  diameters  for  each  pair  of  successive  fixations.  Results  of  the 
experiment  found  a  significant  relationship  between  eye  motion  workload  and  performance 
ratings.  Specifically,  performance  ratings  decreased  as  eye  motion  workload  increased.  Stein 
suggested  that  the  eye  motion  workload  measure  is  more  sensitive  to  changes  in  performance 
than  mean  number  of  fixations  or  saccades  alone.  Overall,  Stein’s  study  provided  support  for  the 
usefulness  of  eye  movements  as  a  measure  of  ATCS  behavior. 
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Stem  et  al.  (1994)  used  eye  movement  measures  and  examined  the  effects  of  time.  They  made 
the  important  point  that  looking  does  not  imply  seeing,  understanding,  or  remembering.  One 
must  extract  information  from  a  display  and  store  that  information  for  later  use.  Stem  and 
colleagues  also  hypothesized  that  missed  signal  detection  may  be  attributed  partly  to  what  they 
termed  gaze  control  inefficiencies  (e.g.,  increases  in  rates  of  eye  blink,  eye  closures,  saccades, 
and  head  movement).  Furthermore,  using  a  hypothesis  similar  to  that  proposed  by  Bills  (1931), 
he  suggested  that  blocks  (i.e.,  microsleep  or  daydreaming)  result  in  attention  being  diverted  away 
from  the  primary  task.  The  operator  must  then  attempt  to  inhibit  attending  to  irrelevant  or 
distracting  parts  of  the  environment  and  maintain  focused  attention.  He  proposed  that  eye 
movements  should  reflect  any  development  of  eye  gaze  inefficiency. 

In  their  experiment.  Stem  et  al.  (1994)  had  the  participants  monitor  a  low-fidelity  radar  display 
simulation.  The  participants  watched  for  untracked  aircraft  (aircraft  without  an  associated  data 
block),  loss  of  altitude  information  fi-om  the  data  block  (inoperative  transponder),  and  separation 
conflicts  (aircraft  at  same  altitude).  They  used  electro-oculography  in  conjunction  with  a  variety 
of  performance  measures.  Results  showed  a  significant  effect  of  time  for  numerous  eye  blink 
measures  such  as  blink  rate,  eye-closing  duration,  50%  window  duration,  blink  flurries,  and 
percent  of  blinks  that  were  part  of  a  flurry.  Additionally,  they  found  a  significant  decrease  in 
saccade  rate  and  an  increase  in  fixation  duration.  All  of  these  effects  supported  the  hypothesis 
than  decrements  in  attention  occur  over  time.  This  present  study  used  measures  related  to  the 
characteristics  of  fixations,  saccades,  blinks,  pupil  size,  and  measures  that  integrate  the  eye 
movement  and  simulator  data. 

3.1.2  Results 


Appendix  H  contains  detailed  information  related  to  visual  scanning  variables  and  analysis 
results.  Section  H.l  presents  the  visual  scanning  variables  (Table  H-1)  and  a  detailed  description 
of  these  variables.  In  Section  H.2,  Tables  H-2  through  H-23  contain  the  full  results  of  the 
inferential  statistical  analyses.  In  Section  H.3,  Tables  H-24  through  H-62  contain  the  results  of 
the  descriptive  statistics. 

3. 1 .2. 1  General  Eve  Movement  Characteristics 

Two  types  of  MANOVAs  examined  changes  in  visual  scanning.  The  first  MANOVA  addressed 
visual  scanning  differences  across  scenarios  and  was  a  2  X  2  (involvement  X  load)  repeated 
measures  analysis.  The  second  MANOVA  addressed  the  differences  across  5-minute  intervals 
and  was  a  2  X  2  X  6  repeated-measures  MANOVA  (involvement  X  load  X  interval).  For  a 
detailed  break  down  of  the  dependent  variables  by  load  and  involvement,  see  Table  2  and 
Appendix  H. 

The  analyses  of  the  eye  movement  data  covered  four  areas.  First,  the  analysis  of  general  eye 
movement  characteristics  involved  the  investigation  of  the  effect  of  the  manipulation  of  the 
independent  variables  on  the  characteristics  of  fixations,  saccades,  blinks,  and  pupil  size.  These 
are  basic  visual  scanning  variables.  Second,  the  analysis  of  fixations  across  scene  planes  focused 
on  how  the  manipulation  of  the  independent  variables  altered  the  number  and  duration  of 
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Table  2.  Variables  Used  to  Assess  General  Eye  Movement  Characteristics 


Variable 

Characteristic 

Tables  in 
Appendix  H 

Saccades 

Duration,  distance,  and  eye  motion  workload 

H-24  to  H-26 

Fixations 

Number,  duration,  area,  and  visual  efficiency 

H-27  to  H-30 

Dwells 

Number,  duration,  and  area 

H-31  to  H-33 

Blinks 

Number  and  duration 

H-34  and  H-35 

Pupil 

Diameter  and  pupil  motion  workload 

H-36 

fixations  for  each  scene  plane.  Third,  the  analysis  of  fixations  across  radarscope  objects  looked 
at  manipulating  the  independent  variables  on  object-based  fixation  characteristics.  Finally,  the 
analysis  of  the  conditional  information  indices  investigated  how  manipulation  of  the  independent 
variables  alters  the  structure  in  the  visual  scan.  Analysis  on  scenario-based  summary  variables 
investigated  the  effects  of  manipulation  of  load  and  involvement,  whereas  5-minute  interval- 
based  analyses  further  investigated  the  effect  of  time. 

General  eye  movement  characteristics  included  variables  without  regard  for  the  scene  plane  or 
object  at  which  the  ATCS  looked  (Table  2).  The  analyses  did  not  include  visual  efficiency,  eye 
motion  workload,  and  pupil  motion  workload  because  an  earlier  study  (Willems  et  al.,  in  press) 
demonstrated  that  these  variables  were  not  sensitive  to  the  level  of  manipulation  used  in  this 
experiment. 

3. 1 .2. 1 . 1  Scenario-Based  Analyses 

Using  the  saccade  duration  and  distance  and  the  fixation  number,  duration,  and  distance,  the 
results  of  the  MANOVA  indicated  that  increasing  load  significantly  altered  the  eye  movement 
characteristics  (Table  H-2).  None  of  the  individual  dependent  variables  related  to  eye 
movements  showed  a  significant  effect  of  load  or  involvement  manipulation  (Table  H-3). 
However,  this  is  applied  research.  Using  ap  <  .01  and  MANOVA  is  a  very  conservative 
approach.  Many  researchers  prefer  going  directly  to  ANOVAs  and  using  p  <  .05  as  the  region 
for  rejecting  the  null  hypothesis.  This  may  produce  more  significant  findings,  which  reflect 
Type  I  error,  but  it  lowers  the  risk  of  missing  significant  differences  that  should  be  addressed 
(Type  n  error).  These  are  treated  below  as  trends  or  indicators,  whereas  others  may  interpret 
them  as  significant  differences. 

At  an  alpha  level  of  p  <  .05,  the  saccade  duration  showed  a  trend  towards  an  increase  under  HL, 
A  conditions  (Figure  3).  See  Table  H-24  for  a  detailed  breakdown  of  saccade  duration  by  load 
and  involvement. 

The  changes  in  load  and  involvement  affected  none  of  the  other  general  eye  movement 
characteristics  (p  <  .01).  Tables  H-24  through  H-33  present  a  detailed  breakdown  of  saccade 
duration  and  distance,  eye  motion  workload,  fixation  number,  duration,  and  area,  visual  scanning 
efficiency,  and  dwell  number,  duration,  and  area  by  load  and  involvement.  Note  that  the 
analyses  only  included  saccade  duration  and  distance  and  fixation  number,  duration,  and  area. 
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Figure  3.  Means  and  standard  deviations  of  saccade  duration  by  load  and  involvement. 

The  second  MANOVA  indicated  that  involvement  significantly  affected  the  variables  often 
associated  with  workload  and  cognitive  activity  (Table  H-4).  To  maintain  an  overall  alpha  level 
of  p  <  .05,  the  adjusted  alpha  level  was  p  <  .017. 

None  of  the  individual  dependent  variables  showed  an  effect  of  the  load  or  the  involvement 
manipulation  (Table  H-5).  Tables  H-34  through  H-36  present  a  detailed  breakdown  of  blink 
number  and  duration  and  pupil  diameter  by  load  and  involvement. 

3. 1 .2. 1 .2  Interval-Based  Analyses 


MANOVAs  on  interval  summary  variables  investigated  the  effect  of  time.  The  MANOVAs 
focused  on  fixation,  saccade,  and  blink  and  pupil  related  variables,  respectively.  For  a  detailed 
break  down  of  the  dependent  variables  by  load,  involvement,  and  time,  see  Tables  H-24  through 
H-43. 

The  MANOVA  on  fixation-related  variables  included  fixation  number,  duration,  and  area  and 
indicated  that  time  significantly  affected  fixation  characteristics  (Table  H-6).  With  three 
dependent  variables  used  in  the  multivariate  analysis,  the  adjusted  alpha  level  to  maintain  an 
overall  alpha  level  of  .05  was  .017. 

The  subsequent  ANOVAs  indicated  that  time  significantly  affected  all  fixation-related  variables 
used  in  the  multivariate  analysis  (Table  H-7).  There  was  a  trend  visible  for  the  interaction 
between  the  effects  of  load  and  time  on  the  number  of  fixations.  The  fixation  duration  showed  a 
trend  towards  an  effect  of  load  and  marginally  for  an  interaction  between  the  effects  of 
involvement  and  time.  The  following  paragraphs  discuss  the  effects  of  time  in  more  detail. 

Time  significantly  affected  the  number  of  fixations  [F(l,  15)  =  13.825,  p  <  .01)].  Tukey’s  post 
hoc  HSD  test  revealed  that  the  number  of  fixations  during  the  first  5  minutes  of  the  simulations 
was  significantly  higher  than  during  subsequent  intervals.  There  was  a  trend  towards  an 
interaction  between  the  effects  of  load  and  time.  Figure  4  shows  that  the  number  of  fixations 
depends  on  time. 
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Figure  4.  Number  of  fixations  by  time  and  load. 

The  time  also  significantly  affected  the  fixation  duration  [F(l,  15)  =  19.004,  p  <  .01].  Tukey’s 
post  hoc  HSD  test  indicated  that  the  fixation  duration  was  significantly  shorter  during  the  first  5 
minutes  of  the  simulations.  There  was  a  trend  towards  an  interaction  between  the  level  of 
involvement  and  the  time  (Figure  5).  The  fixation  durations  were  longer  during  monitoring  than 
during  active  control  in  all  but  the  first  5-minute  interval. 
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Figure  5.  Fixation  duration  by  involvement  and  time. 
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Finally,  time  affected  the  fixation  area  significantly  [F(l,  15)  =  7.496,  p  <  .01].  Post  hoc  Tukey 
HSD  tests  showed  that  fixations  were  more  stable  (as  indicated  by  a  smaller  area  covered  due  to 
small  eye  movements)  in  the  first  5  minutes  of  the  scenarios  than  in  subsequent  5-  minute 
intervals  (Figure  6). 
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Figure  6.  Fixation  area  by  time. 

The  MANOVA  on  saccade  characteristics  included  saccade  duration  and  distance  and  indicated 
an  interaction  of  the  effects  of  load,  involvement  and  time.  It  is  of  little  practical  value,  however, 
to  describe  the  simple  effects  of  the  3-way  interaction.  Due  to  the  3-way  interaction,  one  should 
investigate  the  simple  main  effects  and  the  simple  2-way  interactions.  To  study  simple  effects, 
one  holds  one  of  the  independent  variables  at  a  constant  level  and  looks  at  the  main  effects  and 
the  2-way  interactions  of  the  other  independent  variables.  The  reason  for  investigating  the  5- 
minute  intervals  is  to  look  at  the  time  dependency  of  the  effects  of  the  two  main  independent 
variables,  load,  and  involvement,  on  the  dependent  variables.  The  analysis  of  simple  effects  of 
time  investigated  the  time  dependency  under  each  of  the  four  conditions  involving  load  and 
involvement.  The  effect  of  time  was  significant  under  the  LL,  A  condition  only.  Load 
manipulation  significantly  affected  saccade  characteristics  during  intervals  3  through  5,  whereas 
involvement  had  an  effect  during  intervals  2  through  4  (Table  H-8).  To  maintain  an  overall 
alpha  level  of  .05  for  the  ANOVAs  on  saccade  duration  and  distance,  the  adjusted  alpha  level 
was  .025. 

A  3-way  interaction  existed  for  saccade  duration  (Table  H-9).  It  is  of  little  practical  use  to 
describe  the  simple  effects  of  the  3-way  interaction,  and  we  focused  on  the  simple  effects  of  time 
(Figure  7  and  Figure  8).  The  effect  of  time  was  significant  during  the  LL,  A,  and  the  HL,  M 
conditions.  Load  manipulation  significantly  affected  the  saccade  duration  during  intervals  3  and 
5.  Manipulation  of  involvement  affected  the  saccade  duration  during  intervals  2  through  4  and 
interval  6.  Saccade  durations  were  longer  on  average  for  A  condition  during  those  segments. 
Table  H-51  displays  a  detailed  breakdown  of  the  values  of  saccade  duration  by  load, 
involvement,  and  time. 
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Figure  7.  Saccade  duration  by  time  and  load 
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Figures.  Saccade  duration  by  time  involvement. 

Only  time  affected  saccade  distance  (Figure  9,  Table  H-10).  Mean  saccade  distance  changes 
between  intervals,  but  no  trend  is  visible  by  time.  Keep  in  mind  that  the  saccade  durations  were 
longer  during  several  segments.  It  appears  that  ATCSs  moved  their  eyes  somewhat  slower  when 
actively  controlling  than  when  monitoring  traffic. 
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Figure  9.  Saccade  distance  by  time. 

The  MANOVA  on  blink  and  pupil  characteristics  revealed  that  manipulation  of  the  independent 
variables  did  not  affect  blink  number  and  duration  or  pupil  diameter  (Table  H-1 1). 

The  literature  on  mental  workload  indicates  that  the  number  of  blinks  and  blink  duration  may  be 
indicators  of  the  amount  of  cognitive  activity.  The  current  results  do  not  seem  to  agree  with 
what  other  researchers  have  found.  The  ANOVAs  on  blink  number  and  duration  further 
investigated  the  effect  of  load,  involvement,  and  time.  The  ANOVA  on  blink  duration  did  not 
show  a  significant  effect  of  time.  The  plots  of  blink  duration  by  load  and  time  show,  however, 
that  there  is  a  clear  separation  between  the  means  for  the  two  levels  of  load.  This  separation  is 
visible  for  all  5-minute  intervals  (Figure  10).  Therefore,  although  the  ANOVA  does  not  show  a 
significant  difference  in  blink  duration  due  to  load,  blink  duration  still  may  be  a  valuable 
indicator  of  workload  given  a  large  enough  number  of  the  participants  in  an  experiment. 

3. 1 .2.2  Scene  Planes 

The  introduction  of  this  new  independent  variable  enabled  the  analyses  of  the  effects  of  the 
independent  variables  on  fixation  characteristics  distributed  across  scene  planes.  The  additional 
independent  variable  to  investigate  fixation  characteristics  by  scene  plane  included  eight  levels: 
radarscope,  flight  strip  bay,  keyboard,  track  ball,  ATWIT,  CRD/QAK,  map,  and  landline. 

3. 1 .2.2. 1  Scenario-Based  Analyses 

For  the  scenario-based  analyses,  all  scene  planes  defined  in  the  ATCS  work  environment  formed 
the  levels  of  the  scene  plane  variable.  The  dependent  variables  in  these  analyses  were  the 
number  and  duration  of  fixations.  The  analysis  was  a  2  X  2  X  8  (load  X  involvement  X  scene 
plane)  repeated-measures  MANOVA. 


1  2  3  4  5  6 
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—♦—Low  Task  Load  —■— High  Task  Load 


Figure  10.  Blink  duration  by  load  and  time. 

None  of  the  independent  variables  significantly  affected  the  number  of  fixations  (Table  H-13). 
The  interaction  between  the  effects  of  scene  plane  and  involvement  on  the  fixation  duration  was 
significant  (Table  H-14).  The  simple-effects  analyses  showed  that  involvement  significantly 
increased  the  fixation  duration  for  the  CRD/QAK  and  the  map  [F(l,  15)  =  33.485  and  F(l,  15)  = 
18.707  respectively,  both  at  p  <  .025,  Figure  1 1].  Tables  H-37  and  H-38  display  a  detailed 
breakdown  of  the  number  of  fixations  and  the  fixation  duration  respectively  by  scene  plane  by 
load  and  involvement. 


Figure  11.  Fixation  duration  (msec)  on  the  CRD/QAK  and  the  map  display  by  involvement. 
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3. 1 .2.2.2  Interval-Based  Analyses 

The  analyses  of  the  effects  of  time  and  scene  plane  on  the  number  and  duration  of  fixations  only 
included  the  radarscope  and  flight  strip  bay  as  scene  planes.  The  scenario-based  analysis  had 
already  shown  that  these  two  scene  planes  take  up  92%  of  the  number  of  fixations.  The 
introduction  of  the  time  variable  increases  the  number  of  degrees  of  freedom  needed  for  further 
analysis.  Limitation  of  the  number  of  levels  of  the  scene  plane  variable  enabled  interval-based 
analysis.  The  analysis  was  a2X2X2X6  (load  X  involvement  X  scene  plane  X  interval) 
repeated-measures  MANOV A. 

The  MANOVA  on  the  effect  of  load,  involvement,  time,  and  scene  plane  revealed  a  3-way 
interaction  between  load,  involvement,  and  time  (Table  H-15).  This  interaction  does  not  provide 
further  insight  into  how  the  scene  plane  variable  alters  the  number  and  duration  of  fixations.  The 
investigation,  therefore,  omits  the  analysis  of  this  interaction.  The  MANOVA  results  indicate 
that  the  only  significant  interaction  that  involves  the  scene  plane  variable  is  between  the  effects 
of  the  scene  plane  and  the  time  variables  [A  =  .027,  F(10, 6)  =  21.454,  p  <  .05].  Univariate  , 
analyses  of  the  number  and  duration  of  fixations  also  showed  a  significant  interaction  between 
scene  plane  and  time  (Tables  H-16  and  H-17,  respectively).  To  find  differences  in  fixation 
characteristics  between  scene  planes  is  not  surprising  given  the  fact  that  the  ATCS  priority  is  on 
the  radarscope.  The  ATCS  furthermore  followed  the  experimental  instractions  (i.e.,  the  ATWIT 
device  should  not  interfere  with  controlling  traffic)  and,  therefore,  looked  at  the  ATWTT  device 
only  when  needed.  The  simple  effects  discussed  here  address  the  effect  of  time  per  scene  plane. 
The  interaction  between  the  effects  of  scene  plane  and  time  on  the  number  of  fixations  per  scene 
plane  was  significant  (jP(1,  15)  =  13.036,  p  <  .025). 

3. 1 .2.3  Radar  Scone  Objects 

The  radarscope  objects  included  the  system  area,  other  static  objects,  radar  returns,  and  data 
blocks. 

3. 1 .2.3. 1  Scenario-Based  Analyses 

The  MANOVA  on  the  object-based  fixations  indicated  that  the  load,  involvement,  and  object- 
independent  variables  all  had  significant  main  effects  on  the  fixation  characteristics  (Table  H- 
18).  The  objects  used  were  system  area  (SY),  other  static  objects  (ST),  radar  return  (RR),  and 
data  block  (DB).  The  effects  of  load  and  involvement  were  only  visible  for  the  fixation  duration. 
The  significant  effect  of  object  [A  =  .003,  F(6, 10)  =  587.343,  p  <  .05]  persisted  in  the  univariate 
results.  ANOVAs  on  fixation  number  and  duration  further  investigated  the  effect  of  object  on 
fixation  characteristics. 
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The  number  of  fixations  varied  widely  between  radarscope  objects  (Figure  12).  A  post  hoc 
Tukey  HSD  test  revealed  that  there  was  no  significant  difference  between  the  number  of 
fixations  on  the  system  area  and  other  static  objects.  The  number  of  fixations  on  the  radar 
returns  differed  from  the  number  of  fixations  on  the  data  blocks,  the  system  area,  and  other  static 
objects.  Most  fixations  had  the  radar  return  and  the  data  block  as  their  target.  TheATCSs 
focused  only  few  fixations  on  the  system  area  and  other  static  objects  like  airports  and 
intersections.  They  focused  more  on  data  blocks  than  on  radar  returns. 

Load  significantly  decreased  the  fixations  duration  on  the  radar  scope  objects  [F(l,  15)  =  22.42, 
p  <  .05,  Figure  13].  The  most  pronounced  decrease  in  fixation  duration  was  visible  for  the 
fixations  on  the  systems  area.  Active  control  also  significantly  reduced  the  fixation  duration  on 
radarscope  objects  (Figure  14). 


SY  ST  RR  DB 


Figure  12.  Number  of  fixations  by  radar  scope  object. 


Figure  13.  Fixation  duration  by  radarscope  objects  by  load. 
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Figure  14.  Fixation  duration  by  radarscope  object  by  involvement. 

The  fixation  duration  differed  significantly  depending  on  the  radarscope  object  on  which  the 
ATCS  fixated.  This  result  does  not  come  as  a  surprise.  The  objects  with  most  relevant  and 
complex  information  for  the  ATCS  are  the  radar  return  and  the  data  block.  Figure  15  displays 
the  average  fixation  durations  by  radarscope  object.  The  fixation  duration  on  the  system  area 
shows  a  large  standard  deviation  between  ATCSs.  The  number  of  fixations  on  the  system  area  is 
very  small  in  comparison  to  the  number  of  fixations  on  the  radar  returns  and  data  blocks.  This 
may  explain  some  of  the  variability  of  the  fixation  durations.  The  fixation  durations  on  the  radar 
returns  and  the  data  blocks  are  very  similar  (i.e.,  approximately  500  msec).  A  post  hoc  Tukey 
HSD  test  revealed  that  the  fixation  duration  divided  the  four  objects  into  two  groups.  The  first 
group  consisted  of  the  system  area  and  the  other  static  objects  with  relatively  short  fixations  of 
approximately  200  msec.  The  second  group  consisted  of  the  radar  returns  and  the  data  blocks 
with  an  average  fixation  duration.of  500  msec. 


Figure  15.  Fixation  duration  by  radar  scope  object. 
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3.1.2.3.2  Interval-Based  Analyses 


The  scenario-based  analyses  had  already  shown  that  there  were  few  fixations  on  other  static 
objects  and  the  system.  It  had  also  shown  that  the  duration  of  fixations  on  other  static  objects 
and  the  system  area  were  shorter  than  the  fixations  on  the  radar  returns  and  the  data  blocks.  The 
limited  number  of  fixations  on  other  static  objects  and  the  system  area  would  prevent  a  further 
breakdown  by  time.  We  therefore  used  the  characteristics  of  fixations  on  the  radar  return  and  the 
data  block  as  the  basis  for  the  interval-based  analyses.  Given  the  fact  that  the  aircraft 
representations  carry  most  of  the  information  relevant  to  the  ATCS,  this  seems  a  logical 
restriction. 

We  will  not  discuss  interactions  or  main  effects  that  did  not  involve  the  object  variable  because 
Section  3.1.2. 1.2  presented  these  results.  The  object-based  analysis  of  5-minute  interval  data 
singled  out  fixations  on  radar  returns  and  data  blocks.  The  MANOVA  results  (Table  H-19) 
indicated  that  the  effects  of  load,  involvement,  and  time  on  fixation  characteristics  interacted. 
This  3-way  interaction  did  not  involve  object  variable,  and  we  did  not  address  it  further.  We 
have  discussed  this  3-way  interaction  effect  on  fixation  characteristics  in  Section  3. 1.2. 1.2.  The 
MANOVA  results  further  indicated  two  2-way  interactions.  The  first  interaction  was  between 
load  and  time.  The  other  interaction  involved  the  effects  of  involvement  and  time.  We  did  not 
discuss  these  2-way  interactions  here  because  they  did  not  involve  the  object  variable.  The  main 
effects  of  object  and  time  were  significant. 

The  univariate  analyses  revealed  that  the  main  effect  of  object  was  significant  for  the  number  of 
fixations  [F(l,  15)  =  7.951,  p  <  .05,  Tables  H-20].  No  interactions  that  involved  the  object 
variable  reached  significance.  Section  3. 1 .2. 1 .2  presents  the  effect  of  time  on  general  fixation 
characteristics.  Therefore,  time  did  not  affect  the  number  and  duration  of  fixations  on  radar 
returns  and  data  blocks  differently. 

3. 1.2.4  Stmcture 

The  probability  that  an  ATCS  looks  at  object  B  after  looking  at  object  A  is  an  indication  of  the 
stracture  or  predictability  of  the  visual  scan.  The  transition  probability  from  A  to  B  is  the 
probability  of  looking  at  object  A  followed  by  looking  at  object  B.  These  transition  probabilities 
also  go  by  the  name  of  first  order  Markov  elements.  The  calculation  of  the  conditional 
information  indices  uses  the  probabilities  of  fixations  to  fall  on  two  objects  in  sequence  and 
weighs  this  with  the  proportion  of  fixations  on  these  objects.  The  conditional  information  index 
is  an  indicator  of  the  level  of  stracture  in  the  visual  scan.  The  conditional  information  index  only 
looks  at  a  sequence  involving  two  fixations  at  a  time.  The  indices  will  have  values  that  increase 
when  the  visual  scan  favors  fixations  in  a  certain  order.  Values  closer  to  zero  indicate  less 
stracture  in  the  visual  scan. 

To  investigate  the  existence  of  preferred  sequences  of  objects,  we  calculated  a  conditional 
information  index  based  on  the  object  target  (COB).  To  investigate  the  presence  of  tunnel 
vision,  we  calculated  a  conditional  information  index  based  on  the  distance  between  fixations 
(CRA).  The  probabilities  of  fixations  following  fixations  that  belong  to  the  same  distance  group 
form  the  basis  for  this  measure. 
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Hilbum,  Joma,  Parasuraman,  and  Byme  (1996)  have  used  entropy  in  the  visual  scan  based  on  the 
transition  probabilities  between  areas  on  the  radarscope.  To  investigate  this  approach,  we 
calculated  the  conditional  information  index  based  on  the  location  of  the  center  of  the  fixation  on 
the  radarscope  (CBX). 

Covering  the  entire  airspace  in  the  visual  scan  is  one  of  the  concerns  among  ATCSs.  We 
calculated  the  conditional  information  index  based  on  the  distance  between  the  center  of  the 
fixation  and  the  center  of  the  radarscope  (CRI).  The  CRI  indicates  if  ATCSs  are  more  likely  to 
focus  on  areas  at  equal  distances  from  the  center  of  the  radarscope. 

To  investigate  the  effect  of  load  and  involvement  manipulation,  we  conducted  a  2  X  2  (load  X 
involvement)  repeated  measures  MANOVA.  Depending  on  significant  effects  of  the 
MANOVA,  we  conducted  ANOVAs  on  each  of  the  conditional  information  indices. 

The  MANOVA  showed  that  load  and  involvement  interacted  in  their  effects  on  the  four 
conditional  information  indices  [A  =  .156,  F(4, 12)  =  16.172,  p  <  .05,  Table  H-22.  The 
multivariate  simple  effects  revealed  that  the  effect  of  load  was  significant  independent  of  the 
involvement  level  [A  =  .306,  F(4, 12)  =  6.816  for  monitoring  and  A  =  .143,  F(4, 12)  =  17.934  for 
active  control  respectively,  both  at  p  <  .05].  The  effect  of  involvement  was  only  significant 
under  high  load  conditions  [A  = .  104,  F(4, 12)  =  25.734,  p  <  .05]. 

The  COB  showed  an  interaction  between  the  effects  of  load  and  involvement  (Table  H-23).  We 
therefore  investigated  the  simple  effects  (i.e.,  the  effect  of  load  while  holding  involvement  at 
either  M  or  A  control  and  vice  versa).  The  effect  of  load  on  the  structure  in  the  visual  scan  based 
on  objects  was  significant  under  both  M  and  A  conditions  [F(l,  15)  =  9.947  and  F(l,  15)  = 
76.643  respectively,  both  atp  <  .05].  The  effect  of  involvement  was  only  significant  under  high 
load  conditions  [F(l,  15)  =  24.556, p  <  .05]. 

Figure  16  presents  the  values  for  the  object-based  conditional  information  index.  The  structure 
in  the  visual  scan  decreases  with  an  increase  in  load.  Under  HL  conditions,  A  control  reduces 
the  structure  in  the  visual  scan. 


♦  Low  —•—High 

Figure  16.  Object-based  conditional  information  index  by  load  and  involvement. 
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The  univariate  ANOVA  indicated  that  only  load  had  a  significant  effect  on  the  CRA 
[F(l,  15)  =  12.802, p  <  .05].  With  an  increase  in  load,  the  structure  increased  (Figure  17). 
Although  the  CBX  indicated  that  there  was  more  structure  in  the  visual  scan  when  based  on 
position  on  the  radarscope,  there  was  no  difference  in  the  CBX  due  to  manipulation  of  load  or 
involvement  levels.  The  manipulation  of  load  and  involvement  had  no  effect  on  the  CRI. 


♦—Low  m—  High 


Figure  17.  Range-based  conditional  information  index  by  load  and  involvement. 

3.1.3  Discussion 

Manipulation  of  load  and  involvement  did  not  affect  the  general  eye  movement  characteristics 
significantly.  Although  the  saccades  tended  to  last  longer  with  HL  and  A  control,  this  is  not  in 
correspondence  with  a  saccade  distance.  The  literature  on  saccade  characteristics  suggests  that 
saccade  duration  strongly  correlates  with  saccade  distance.  The  fact  that  we  did  not  find  an 
effect  of  the  manipulation  of  our  independent  variables  on  saccade  distance  makes  the  trend  in 
saccade  duration  suspect. 

Willems  et  al.  (in  press)  were  unable  to  investigate  the  effect  of  time  due  to  a  confounding  effect 
of  traffic  build  up  in  the  first  15  minutes  of  their  simulation  scenarios.  The  current  study 
eliminated  the  confounding  effect  of  traffic  build  up  by  providing  ATCSs  with  traffic  similar  to 
what  they  experience  during  a  relief  briefing.  The  data  analysis  shows  that  time  affects  the  eye 
fixation  characteristics.  During  the  first  5  minutes  of  a  30-minute  scenario,  the  ATCS  has  more 
fixations  that  are  both  shorter  and  more  stable.  A  possible  explanation  for  this  phenomenon  is 
that  a  considerable  amount  of  verbal  information  transfer  takes  place  during  the  relief  briefing. 
The  ATCS  may  not  need  to  retrieve  as  much  information  from  the  radarscope  during  the  first  5 
minutes  than  during  subsequent  intervals  while  building  an  internalized  model  or  picture. 

Time  affected  the  saccade  duration  during  LL,  A  and  HL,  M  conditions  only.  Although  time 
alters  the  saccade  distance,  there  is  no  consistent  trend  visible  towards  an  increase  or  a  decrease 
in  distance. 

The  ATCSs  spent  most  of  their  fixations  on  the  radarscope  and  the  flight  strip  bay.  The  A 
condition  increased  the  duration  of  fixations  on  the  CRD/QAK  and  the  map  significantly.  Given 
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the  priorities  of  the  ATCSs,  it  is  not  surprising  to  find  differences  in  the  number  and  duration  of 
fixations  depending  on  the  scene  plane. 

Most  of  the  fixations  on  the  radarscope  focused  on  the  radar  return  and  the  data  block. 

Increasing  load  resulted  in  shorter  fixations  on  radarscope  objects.  Under  M  conditions,  the 
fixations  were  shorter  than  under  A  conditions.  Willems  et  al.  (in  press)  suggest  that  longer 
fixations  indicate  more  cognitive  processing.  The  results,  therefore,  indicate  that,  under  HL,  less 
processing  takes  place  during  a  fixation  on  an  individual  target  than  under  LL.  Similarly,  under 
A  conditions,  more  processing  takes  place  than  under  M  conditions. 

To  determine  the  stmcture  in  the  visual  scan,  we  have  used  four  indices  derived  from  the 
conditional  information  index.  Ellis  and  Stark  (1986)  first  introduced  the  conditional 
information  index.  This  index  indicates  the  predictability  of  the  visual  scan.  If  the  visual  scan  is 
completely  random,  the  conditional  information  index  is  equal  to  zero.  We  can  see  differences 
between  the  conditional  information  indices  depending  on  what  forms  the  basis  for  the 
calculations.  If  one  calculates  the  transition  probabilities  between  locations  on  the  r^arscope, 
there  seems  to  be  more  structure  in  the  visual  scan.  The  structure  in  the  airspace  is  mostly 
responsible  for  this  result.  It  indicates  that  it  is  very  likely  that  an  ATCS  searches  the  radarscope 
in  a  pattern  based  on  location  on  the  scope  rather  than  sequences  of  aircraft  or  distances  between 
fixations.  There  is,  however,  no  difference  in  the  radarscope  position  based  on  the  conditional 
information  index  between  conditions.  Under  HL,  the  distribution  of  fixations  between 
radarscope  objects  was  more  random  than  under  LL.  The  A  condition  increased  the  randomness 
in  the  visual  scan  only  under  the  HL  conditions.  Under  HL,  ATCSs  were  less  likely  to  follow  a 
pattern  of  fixations  based  on  the  distance  between  fixations. 

3.2  Air  Traffic  Workload  Input  Technique 

3.2.1  Background 

In  this  research,  we  used  the  ATWIT  to  study  the  ATCS  perceived  workload.  Stein  (1985)  first 
introduced  ATWIT,  which  is  an  online  measure  that  requires  ATCSs  to  indicate,  at  set  times, 
their  perception  of  their  current  workload.  ATWIT  is,  Aerefore,  an  instantaneous  probe  that 
investigates  overall  perceived  workload.  Contrary  to  the  NASA  Task  Load  Index  (TLX)  (Hart  & 
Staveland,  1988),  for  example,  the  participants  do  not  need  to  break  down  their  workload  by 
origin.  Another  advantage  of  the  ATWIT  over  post-scenario  ratings  of  workload  is  that  AIWTT 
asks  for  input  during  the  simulation  instead  of  relying  on  ATCS  memory  during  the  scenario. 

3.2.2  Results 

For  the  analyses  of  the  online  workload  measure  used  in  this  study,  we  used  both  the  workload 
rating  and  the  latency.  See  the  tables  in  Appendix  I  for  details  of  these  analyses.  The  latency 
indicates  how  long  it  took  an  ATCS  to  respond  to  the  ATWIT  device.  We  analyzed  ATWIT 
latencies  and  ratings  with  a  2  X  2  X  6  (load  X  involvement  X  interval)  repeated-measures 
MANOVA.  Significant  interactions  were  found  for  load  X  involvement  [F(2, 14)  =  24.65,  p  < 
.05],  load  X  interval  [F(10, 6)  =  6.34,  p<.05],  and  involvement  X  interval  [F(10, 6)  =  15.52,  p  < 
.05].  We  further  investigated  the  significant  interactions  with  an  ANOVA  procedure.  The  first 
set  of  ANOVAs  examined  the  load  X  involvement  interaction  for  ATWIT  latency. 


27 


The  MANOVA  on  ATWIT  rating  and  latency  by  load,  involvement,  and  time  indicated  that  the 
effects  of  the  independent  variables  interacted  in  pairs.  The  3-way  interaction  was  not 
statistically  significant.  Load  only  affected  the  ATWIT  characteristics  under  A  conditions  [A  = 
.148,  F(2, 14)  =  40.359,  p  <  .05,  Table  I-l].  Load  affected  the  ATWIT  characteristics 
throughout  the  six  5-minute  intervals.  Involvement  affected  the  ATWIT  characteristics  under 
both  LL  and  HL  conditions  [A  =  .442,  F(2,  14)  =  8.837  and  A  = .  13 1 ,  F(2, 14)  =  46.296 
respectively,  both  atp<  .05,  Table  I-l].  Involvement  also  affected  the  ATWIT  characteristics 
throughout  the  six  5-minute  intervals. 

Figure  18  and  Table  1-4  present  the  means  and  SDs  of  ATWIT  ratings  across  load,  involvement, 
and  time  levels.  All  2-way  interactions  were  significant  for  the  ATWIT  rating.  We  used  simple 
effects  to  investigate  when  ATWIT  ratings  differed.  Although  the  3-way  interaction  was  not 
significant.  Table  1-4  provides  a  breakdown  of  the  ATWIT  ratings  by  conditions. 


Figure  18.  ATWIT  ratings  by  load,  involvement,  and  time. 

Increasing  load  caused  an  increase  in  the  average  ATWIT  rating  under  both  M  and  A  conditions 
[F(l,  15)  =  6.882  and  F(l,  15)  =  74.447  respectively,  both  atp  <  .05,  Table  1-2  and  Figure  19]. 
Active  condition  scenarios  received  higher  ATWIT  ratings  than  monitoring  conditions  under  LL 
and  HL  conditions  [F(l,  15)  =  18.855  andF(l,  15)  =  95.018  respectively,  both  atp  <  .05].  The 
increase  in  workload  estimates  due  to  an  increase  in  load  was  higher  under  A  conditions  than  M 
conditions. 


Figure  19.  ATWIT  ratings:  Means  and  SDs  by  load  and  involvement. 

The  effect  of  load  on  the  ATWIT  rating  interacted  with  the  time  variable  [F(l,  15)  =  4.900,  p  < 
.05,  Table  1-2].  For  all  intervals,  an  increase  in  load  led  to  an  increase  in  perceived  workload 
(Figure  20).  As  the  figure  indicates,  there  was  a  large  variability  in  ratings  between  ATCSs.  It 
also  clear  that  the  ATCSs  felt  that  the  scenarios  were  causing  only  moderate  workload. 


Figure  20.  ATWIT  ratings:  Means  and  SDs  by  load  and  time. 


The  effect  of  involvement  and  time  interacted  [F(l,  15)  =  13.180,  p  <  .05,  Table  1-2].  ATCSs 
rated  the  perceived  workload  higher  under  A  conditions.  Under  M  conditions,  the  workload 
remained  constant  over  time.  Under  A  conditions,  the  workload  slowly  increases  over  time 
(Figure  21). 
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—♦—Monitoring  —■—Active  Control 

Figure  21 .  ATWIT  ratings:  Means  and  SDs  by  involvement  and  time. 

Only  involvement  had  a  significant  effect  on  the  ATWIT  latency  [F(l ,  15)  =  6.574,  p  <  .05, 
Table  1-3].  The  ATCSs  took  longer  to  respond  to  the  ATWIT  under  A  conditions  than  under  M 
conditions  (Figure  22). 


Figure  22.  ATWIT  latency  by  involvement  and  time. 


3.2.3  Discussion 


ATCSs  estimated  their  workload  every  5  minutes.  The  instructions  to  the  participants  were  very 
specific  as  to  how  we  expected  them  to  respond,  emphasizing  that  the  workload  estimate  should 
be  instantaneous.  The  instructions  also  reinforced  that  estimated  workload  was  not  equivalent  to 
load.  The  instructions  provided  the  participants  with  clear  anchors  for  several  levels  of 
workload,  all  related  to  being  able  to  complete  the  tasks  at  hand. 

The  effect  of  increasing  load  and  changing  the  level  of  involvement  interacted  with  time.  The 
participants  indicated  that  their  workload  was  higher  under  HL.  Perceived  workload  was  also 
higher  under  A  conditions.  The  ATCSs  did  not  perceive  the  HL  as  producing  high  workload. 
Even  for  die  HL,  A  scenario,  the  average  ATWIT  rating  was  approximately  6  on  a  10-point 
scale.  During  the  development  of  the  simulation  scenarios,  the  SME  had  indicated  that  this 
scenario  would  produce  a  high  workload.  There  are  at  least  two  possible  explanations  for  this 
result.  First,  ATCSs  often  underestimate  their  workload.  The  ATCSs  have  a  “can  do”  attitude 
that  has  helped  them  survive  in  the  current  ATC  system.  Underestimation  may  have  contributed 
to  the  lower  than  expected  workload  estimates.  The  ATWTT  ratings  indicated  that  ATCSs  only 
perceived  a  moderate  workload.  A  second  explanation  may  lie  in  the  composition  of  the  generic 
en  route  airspace  that  we  used  in  this  experiment.  To  allow  ATCSs  to  familiarize  themselves 
with  the  airspace  quickly,  we  built  a  simple  airspace.  Although  our  SME  indicated  that  the  load 
was  high,  this  may  have  related  to  the  level  of  traffic  more  than  expected  workload  due  to  the 
combined  airspace  and  traffic  load. 

Under  M  conditions,  the  estimated  workload  was  constant  over  time.  Under  A  conditions,  the 
estimated  workload  slowly  increased  over  time.  This  result  would  favor  the  M  conditions 
because  it  seems  to  eliminate  the  effect  of  time  on  perceived  workload. 

3.3  Situation  Presence  Assessment  Method 

3.3.1  Background 

Unlike  eye  movements  in  ATC,  a  considerable  amount  of  research  effort  has  recently  focused  on 

5  A  in  dynamic  systems.  Although  varied  definitions  have  been  proposed  to  capture  the  essence 
of  SA  (Endsley,  1988;  Fracker,  1989;  Mogford,  1994;  Pew,  1994),  Aere  is  currently  no  agreed 
upon  definition.  Tolk  and  Keether  (1982)  thought  of  it  as  the  ability  to  envision  the  current  and 
future  disposition  of  both  red  and  blue  aircraft  and  surface  threats.  Endsley’ s  definition  of  SA  is 
more  general  and  widely  cited:  “...the  perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status 
in  the  near  future”  (p.  3).  Regardless  of  which  definition  is  used,  most  researchers  agree  that  the 
perception  and  understanding  of  elements  in  the  present  situation  is  an  important  process  in 
maintaining  SA.  Furthermore,  one  must  use  this  information  to  predict  and  anticipate  future 
events. 

The  researchers  have  also  used  many  different  methods  to  measure  how  operators  develop  and 
maintain  SA.  The  gamut  of  SA  measures  includes  both  subjective  and  objective  techniques. 
Previously  employed  measures  include  physiological  measures  such  as  eye  movements  (Moray 

6  Rotenberg,  1989;  Wierwille  &  Eggemeier,  1993),  verbal  protocol  analysis  (Ohnemus  &  Biers, 
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1993;  Sullivan  &  Blackman,  1991),  retrospective  recall  (de  Groot,  1965;  Kibbe,  1988),  rating 
techniques  (Reid  &  Nygren,  1988;  Taylor,  1990),  memory  probes  (Endsley,  1988),  and  on-line 
queries  (Durso  et  al.,  1995).  Most  of  these  techniques  have  demonstrated  some  degree  of 
validity  and  usefulness. 

The  current  experiment  took  place  within  the  realm  of  a  high  fidelity,  simulated  ATC 
environment.  The  technique  to  assess  SA  was  the  SPAM  (Durso  et  al.,  1995).  SPAM  provided 
a  means  to  assess  S  A  without  dismpting  or  otherwise  significantly  changing  the  ATC  task  as 
performed  in  the  field.  Initially  validated  in  an  experiment  using  chess  players  as  the 
participants,  SPAM  allowed  the  presentation  of  queries  using  a  landline.  Thus,  the  participants 
answered  queries  in  the  SPAM  just  as  they  would  when  coordinating  activities  between  their 
sector  and  other  adjacent  sectors  or  facilities. 

SPAM  does  not  require  freezing  or  stopping  the  scenario  to  collect  data.  Researchers  have 
criticized  techniques  that  assess  SA  by  freezing  the  simulation  like  the  Situation  Awareness 
Global  Assessment  Technique  (SAGAT)  (Endsley,  1988)  for  its  intrasiveness  and  possible  task- 
altering  qualities  (Sarter  &  Woods,  1991).  Furthermore,  such  techniques  use  memory  probes 
that  require  the  participant  to  recall  information  to  provide  a  response.  Proportion  correct  serves 
as  the  dependent  measure  in  memory  probes  like  SAGAT.  On  the  other  hand,  the  SPAM 
technique  allows  the  participants  to  use  all  information  available  to  them  because  it  does  not 
freeze  the  scenario.  Rather  than  assessing  memory  in  and  of  itself,  SPAM  assesses  the 
participant  ability  to  find  or  extrapolate  information  from  the  environment  and,  hence,  response 
time  (RT)  is  the  dependent  measure.  The  distinction  between  SAGAT  and  SPAM  is  an 
important  one,  especially  when  considering  tasks  where  memory  for  verbatim  information  is  not 
critical  and  may  be  detrimental  to  performance  (Bisseret,  1971;  Gronlund  et  al.,  1996). 

3.3.2  Results 


We  conducted  three  separate  analyses  on  the  data  collected  from  the  SPAM.  See  the  tables  in 
Appendix  J  for  details  of  these  analyses.  The  first  analysis  examined  the  time  it  took  the 
participants  to  answer  the  ringing  landline.  This  landline  latency  measure  served  as  a  secondary 
workload  probe.  We  investigated  the  effect  of  the  independent  variables  and  the  type  of  question 
with  a  2  X  2  X  2  (load  X  involvement  X  question  type)  repeated-measures  ANOVA. 

The  ANOVA  resulted  in  a  significant  load  X  involvement  interaction  [F(l,  47)  =  17.47, p  <  .05]. 
There  was  no  effect  of  either  load  or  question  type.  Simple-effects  ANOVAs  revealed  that  the 
load  X  involvement  interaction  was  due  to  load  within  the  A  condition  [F(l,  47)  =  15.91,/?  < 

.05]  and  involvement  within  the  HL  condition  [F(l,  47)  =  87.52,  p  <  .05,  Table  J-1]. 

The  second  analysis  concerned  the  time  it  took  us  to  query  the  participant.  We  conducted  a  2  X 
2X2  (load  X  involvement  X  question  type)  repeated-measures  ANOVA  to  ensure  that  queries 
were  of  equal  length  in  all  conditions.  We  found  no  significant  effects  (Table  J-2).  Therefore, 
the  mean  length  of  the  queries  was  equivalent  during  all  conditions.  This  finding  is  important 
because  it  suggests  that  the  participants  did  not  have  more  or  less  time  to  consider  a  query  during 
any  particular  condition. 
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The  third  analysis  of  the  SPAM  data  addressed  the  main  intent  of  the  SPAM  measure  (e.g.,  to 
determine  the  quality  of  SA  under  various  conditions).  Reaction  time  to  answer  the  SPAM 
showed  a  significant  3- way  interaction  between  load,  involvement,  and  question  type  [F(l,  47)  = 
12.75,  all  at  p  <  .05,  Table  J-3].  To  interpret  the  results,  we  investigated  simple  effects  broken 
down  by  type  of  question. 

The  simple-effects  analysis  of  the  present  questions  indicated  an  interaction  between  load  and 
involvement.  To  investigate  the  effects  on  the  RT,  we  conducted  simple-simple  analyses  where 
we  dealt  with  three  independent  variables.  We  held  the  first  independent  variable  (the  type  of 
question)  constant.  Subsequently,  we  held  a  second  independent  variable  (load  or  involvement) 
constant  and  looked  at  the  effect  of  the  third  independent  variable.  The  results  indicated  that,  for 
the  present  questions,  the  effect  of  load  was  only  significant  under  M  conditions  [F(l,  47)  = 
20.568,  p  <  .05,  Table  J-4].  The  effect  of  involvement  was  only  significant  under  HL  [F(l,  47)  = 
26.847,  p<. 05,  Table  J-4. 

The  HL,  M  condition  drives  the  3-way  interaction  (Figure  23).  The  result  suggests  that  the 
participants  maintained  an  equal  level  of  SA  in  all  conditions  except  one.  When  the  participants 
were  monitoring  a  busy  scenario,  they  had  relatively  worse  SA  for  present  information.  In  fact, 
it  took  the  participants  twice  as  long  to  answer  queries  about  present  information  under  the  HL, 
M  condition  than  under  the  other  three  conditions.  The  simple-effects  analysis  of  the  future 
questions  revealed  no  effects  of  the  independent  variables  on  the  RT  (Table  J-5). 


—♦—Low  Load  —■—High  Load 


Figure  23.  SPAM  response  time  to  present  questions  by  load  and  involvement. 
3.3.3  Discussion 


The  analyses  of  time  to  answer  the  landline  indicated  that  the  mean  RT  to  answer  the  landline  in 
A  conditions  was  longer  than  in  M  conditions.  In  A  conditions,  mean  RT  to  answer  the  landline 
was  longer  when  load  was  high.  Load  did  not  affect  mean  RT  to  answer  the  landline  under  M 
conditions.  Involvement  did  not  have  an  effect  on  mean  RT  to  answer  the  landline  in  LL 
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conditions  but  did  have  an  effect  in  HL  conditions.  Under  HL,  mean  RT  was  longer  when  the 
participant  was  actively  controlling  the  scenario.  Assuming  that  mean  RT  to  answer  the  landline 
increases  with  workload,  the  results  support  the  finding  that  workload  was  higher  during  A 
conditions  than  during  M  conditions.  Additionally,  the  results  support  the  effectiveness  of  load 
manipulations  within  the  A  condition  in  that  mean  RT  was  longer  under  HL  than  under  LL. 

We  found  no  significant  changes  in  the  mean  length  of  the  queries.  This  finding  is  important 
because  it  suggests  that  the  participants  did  not  have  more  or  less  time  to  consider  a  query  during 
any  particular  condition.  The  results  suggest  that  active  participation  is  important  for 
maintaining  SA  when  load  (e.g.,  scenario  complexity)  is  high. 

3.4  Real-Time  Objective  Performance 

3.4.1  Background 

In  response  to  the  need  for  new  tools  to  evaluate  proposed  changes  to  the  ATC  system,  the  FAA 
has  developed  methods  and  measurements  in  real-time  ATC.  The  DR&A  program  incorporates 
the  calculations  of  most  of  the  variables  presented  by  Buckley,  DeBaryshe,  Hitchner,  &  Kohn 
(1983). 

3.4.2  Results 

This  experiment  tested  the  effect  of  two  levels  of  involvement  on  ATCS  performance  and 
behavior.  See  the  tables  in  Appendix  K  for  detailed  descriptions  of  the  analysis. 

During  A  conditions,  the  RTOP  variables  are  an  indication  of  ATCS  performance  related  to 
conflicts,  complexity,  handoff  efficiency,  and  communications.  For  the  simulation  scenarios 
used  for  the  M  conditions,  we  had  recorded  traffic  controlled  by  the  SME.  ATCSs  observed 
these  scenarios  and  answered  the  landline  but  did  not  communicate  with  the  simulation  pilots  nor 
did  they  need  to  interact  with  the  PVD  and  CRD.  The  RTOP  variables  under  M  conditions, 
therefore,  merely  reflected  the  performance  of  the  SME  that  had  controlled  the  recorded  traffic. 
The  comparison  of  the  RTOP  variables  across  involvement  levels  would  result  in  comparing  the 
participant  performance  with  the  SME  performance.  Because  the  intent  of  this  experiment  was 
to  compare  performance  and  behavior  of  the  same  ATCS  across  conditions,  we  limited  the 
analyses  to  the  comparison  between  load  levels  for  the  A  condition  only.  We  investigated  the 
effect  of  load  under  the  A  condition  on  a  subset  of  the  RTOP  variables.  The  variables  included 
in  the  analysis  consisted  of  three  categories:  PTT,  aircraft  changes,  and  distance  and  time  under 
control.  To  obtain  information  about  how  load  affected  ATCS  actions  per  aircraft,  we  calculated 
the  total  number  of  a  particular  type  of  action  divided  by  an  estimate  of  the  total  number  of 
aircraft  handled  by  the  ATCS.  For  example,  the  number  of  altimde  changes  is  calculated  as  the 
total  number  of  altitude  changes  made  to  an  aircraft  under  control  of  ATCS  A  plus  the  number  of 
changes  made  under  control  of  ATCS  B.  This  is  then  divided  by  the  number  of  aircraft  handled 
by  ATCS  B.  In  this  manner,  we  were  able  to  circumvent  the  problem  of  finding  trivial  results 
due  to  the  changes  made  in  the  number  of  aircraft  in  the  airspace  to  change  the  load. 
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The  results  indicated  that  the  changes  made  to  an  aircraft  flight  path  differed  significantly 
between  the  LL  and  HL  levels  [A  =  .074,  F(2, 14)  =  87.291,  p  <  .05,  Table  K-2].  The  univariate 
analysis  of  the  number  of  altitude,  heading,  and  speed  changes  per  aircraft  showed  that  only  the 
number  of  altitude  changes  per  aircraft  increased  significantly  with  load  [F(l,  15)  =  14.352,  p  < 
.05,  Table  K-3]. 

3.4.3  Discussion 

After  correction  for  the  number  of  aircraft  handled  by  the  ATCSs,  there  were  only  minimal 
differences  in  variables  derived  from  the  DRA  between  low  and  high  load  conditions.  The  load 
increase  resulted  in  an  increase  in  the  number  of  altitude  changes  per  aircraft.  ATCSs  use  more 
control  instructions  per  aircraft  to  move  aircraft  through  their  airspace  when  load  increases.  It 
seems  that  the  increase  in  load  affects  ATCS  ability  to  plan.  ATCSs,  therefore,  need  to  use  more 
control  instruction  to  maintain  a  safe  and  expeditious  flow  of  traffic. 

3.5  Subject  Matter  Expert  Ratine  Forms 

3.5.1  Background 

In  our  simulations,  we  use  subject  matter  expertise  and  knowledge  to  evaluate  the  performance 
of  participating  ATCSs.  To  record  the  evaluations,  we  used  an  OTS  rating  technique  developed 
at  Ae  RDHFL.  Several  other  studies  have  used  the  OTS  form  successfully  (e.g.,  Guttman  et  al., 
1995;  Sollenberger  &  Stein,  1995).  We  adapted  the  rating  form  for  easier  use  by  the  SME. 
SMEs  in  our  study  used  a  form  that  contained  rating  items  and  anchors  and  a  separate  comment 
sheet.  They  received  training  on  how  to  use  the  evaluation  form  and  how  to  anchor  their  ratings. 

3.5.2  Results 

The  following  descriptive  summary  provides  an  overview  of  the  observer  data.  Because  the  M 
conditions  provided  little  observable  behavior  for  an  SME  to  anchor  the  ratings,  we  did  not 
require  them  to  fill  out  a  rating  form  for  these  conditions.  The  tables  in  Appendix  L  provide  the 
means  and  standard  deviations  for  the  rating  form  ratings  by  load. 

3.5.2. 1  Providing  ATC  Information 

Load  reduced  the  OTS  rating  of  Providing  ATC  Information.  All  three  elements  of  the  ATC 
information  section  showed  a  lower  OTS  rating  for  the  LL  conditions  (Figure  24). 


35 


Maintaining  Ensuring  Detecting  Pilot  Correcting 
Situation  Positive  Control  Deviations  from  Errors  in  a 
Awareness  Control  Timely  Manner 


Instructions 

□Low Load  ■  High  Load 


Figure  24.  Providing  ATC  information  by  load. 

3.5.2.2  Prioritizing 

The  SMEs  rated  items  related  to  Prioritizing  lower  with  an  increase  in  load  (Figure  25).  The 
results  showed  that  ATCSs  better  organized  their  actions  in  order  of  importance  under  LL. 

Raters  perceived  that  ATCSs  preplanned  control  actions  less  under  HL.  ATCSs  handled  control 
tasks  for  several  aircraft  better  under  LL  conditions.  With  an  increase  in  load,  ATCSs  flight  strip 
marking  decreased. 


Taking  Actions  Preplanning  Handling  Control  Marking  Flight 


in  an  Control  Actions  Tasks  for  Several  Strips  while 
Appropriate  Aircraft  Peifonning  Other 

Order  of  Tasks 

Importance 

□  Low  Load  ■  Hgh  Load 


Figure  25.  Prioritizing  by  load. 
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3.5.2.3  Attention  and  SA 


The  SMEs  indicated  that  all  items  related  to  ATCSs  Attention  and  SA  were  lower  under  HL 
conditions  than  under  LL  conditions  (Figure  26). 


Maintaining  Ensuring  Detecting  Pilot  Correcting 
Situation  Positive  Control  Deviations  from  Errors  in  a 
Awareness  Control  Timely  Manner 


Instructions 
□Low  Load  ■  High  Load 


Figure  26.  Attention  and  situation  awareness  by  load. 
3.5.2.4  Safe  and  Efficient  Traffic  Flow 


The  SMEs  rated  the  items  related  to  Safe  and  Efficient  Traffic  Flow  lower  under  HL  conditions 
(Figure  27).  At  first  glance,  it  seems  that  ATCSs  efficient  sequencing  of  arrival  and  departure 
aircraft  does  not  change  with  an  increase  in  load. 


Detecting  Pilot  Correcting  Errors  Maintaining 
Deviations  from  in  a  Timely  Separation  and 
Control  Manner  Resolving 

Instructions  Potential 

Conflicts 


Sequencing 
Arrival  and 
Departure 
Aircraft 
Efficiently 


□Low  Load  ■  High  Load 


Figure  27.  Safe  and  efficient  traffic  flow  by  load. 
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3.5.2.5  Communications 

The  SMEs  rated  most  items  related  to  Communications  lower  under  HL  conditions  (Figure  28). 
There  was  a  trend  visible  for  a  reduction  in  how  clear  ATCSs  communicated  with  an  increase  in 
load.  Load  also  reduced  how  well  ATCSs  listened  to  pilot  readbacks  and  requests.  The  increase 
in  load  did  not  seem  to  affect  the  use  of  proper  phraseology. 


0  -I 


Using  Control  Using  Proper  Communicating  Listening  to 

Instructions  Phraseology  Clearly  and  Pilot  Readbacks 
Effectively  Efficiently  and  Requests 

□Low  Load  IHighLoad 


Figure  28.  Communications  by  load. 


An  increase  in  load  affected  all  items  related  to  Technical  Knowledge  (Figure  29).  The  SMEs 
rated  items  on  knowledge  of  LOAs  and  SOPs  and  aircraft  capabilities  and  limitations  lower  with 
increase  in  load.  They  indicated  that  ATCSs  used  the  equipment  less  effectively  with  an  increase 
in  load. 


Showing  Knowledge  Showing  Knowledge  Showing  Effective 
of  LOAs  and  SOPs  of  Aircraft  Use  of  Equipment 

Capabilities  and 
Limitations 

□Low  Load  ■  High  Load 
Figure  29.  Technical  knowledge  by  load. 
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3.5.3  Discussion 

Both  SMEs  found  that  ATCSs  provided  adequate  ATC  information  under  both  load  conditions. 
Although  the  formal  analyses  did  not  show  an  effect  of  the  manipulation  of  load,  the  data 
showed  a  tendency  towards  lower  ratings  of  the  provision  of  information  under  HL.  The  ATCSs 
seem  to  compromise  the  quality  of  the  information  provided  to  pilots  and  other  ATCSs  under 
HL.  Providing  this  information  requires  the  allocation  of  some  of  the  ATCS  resources.  The 
ATCSs  rated  the  simulations  to  have  only  moderate  levels  of  workload.  Under  higher  workload 
conditions,  the  ATCS  information  services  may  suffer  more  seriously. 

Although  the  information  provided  to  pilots  and  other  ATCSs  did  not  suffer  from  an  increase  in 
load,  ATCSs  prioritization  did.  They  organized  their  actions  in  a  way  that  conformed  less  to  the 
level  of  action  priority.  This  finding  went  hand  in  hand  with  a  decrease  in  the  quality  of 
preplanning  of  control  actions  with  an  increase  in  load.  It  is  likely  that,  due  to  a  bre^  down  in 
maintaining  the  bigger  picture,  ATCSs  were  less  efficient  in  preplanning  their  control  actions. 
The  loss  of  efficiency  in  preplanning  control  actions  in  turn  may  have  led  to  not  executing 
control  actions  in  order  of  priority. 

The  rating  form  data  indicated  that  the  increase  in  load  led  to  a  reduction  in  S  A.  Increasing  load 
seems  to  affect  the  ATCS  ability  to  see  the  bigger  picture,  or  it  causes  them  to  be  less  aware  of 
the  developing  situation.  In  fact,  under  HL,  both  raters  indicated  that  the  ATCSs  had  less  than 
average  SA.  This  occurred  during  scenarios  that,  according  to  the  ATCSs,  caused  only  moderate 
workload.  The  SMEs  perceived  lliat  ATCSs  corrected  errors  less  well  under  HL. 

An  ATCS  primary  responsibility  is  to  maintain  safe  and  efficient  traffic  flow  at  all  times.  The 
rating  form  data  in  the  current  study  indicated  that  the  participants  did  this  for  both  levels  of 
load. 

Our  formal  analyses  showed  no  difference  in  the  quality  of  communications  as  rated  by  the 
SMEs  by  load.  There  was,  however,  a  trend  towards  a  reduction  in  the  quality  of 
communications  because  of  an  increase  in  load. 

The  SMEs  perceived  a  reduction  in  Technical  Knowledge  with  an  increase  in  load.  It  is  not 
likely  that  the  ATCSs  actually  had  less  technical  knowledge.  Instead,  it  is  more  likely  that  under 
HL,  the  ATCSs  were  less  able  to  apply  this  knowledge. 

Overall,  observers  rated  performance  somewhat  lower  under  HL.  This  is  a  common  finding  and 
may  well  reflect  a  component  of  observer  expectations  and  possible  true  variance  of  lower 
performance  under  HL.  It  is  not  possible  to  separate  these  components  at  this  time. 

3.6  Recall 

3.6.1  Background 

The  transition  to  Free  Flight  can  also  affect  representation  in  memory.  As  suggested  by  Hopkin 
(1988),  the  lack  of  active  participation  can  have  adverse  effects  on  the  maintenance  of 
information  in  memory.  Several  studies  have  examined  the  role  of  memory  in  ATC.  Bisseret 
(1971)  examined  ATCS  recall  across  various  levels  of  expertise  and  load.  Results  indicated  that 
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future  states  of  aircraft  were  an  important  aspect  of  the  ATCSs  memory  representations. 

Evidence  for  implicit  momentum  OFinke,  Freyd,  &  Shyi,  1986;  Finke  &  Shyi,  1988)  was  found. 
ATCSs  recalled  aircraft  position  as  being  forward  of  the  actual  position.  In  addition  to  the 
importance  of  future  states,  recall  errors  provided  evidence  that  ATCSs  stored  gist  (relative) 
information  in  memory  as  opposed  to  verbatim  information.  Gronlund  et  al.  (1996)  found  that, 
whereas  ATCSs  were  not  very  good  at  recalling  specific  information  such  as  altitude  and  speed, 
they  were  able  to  correctly  recall  the  relational  associations  between  aircraft.  For  example, 
although  the  participants  could  not  remember  the  exact  altitude  of  aircraft  A  or  B,  they  knew  that 
aircraft  A  was  higher  or  not  at  the  same  altitude  as  aircraft  B  . 

Gronlund  et  al.  (1996)  also  examined  how  ATCSs  represent  information  in  memory  by  looking 
for  evidence  of  “chunking.”  In  a  procedure  similar  to  that  used  by  de  Groot  (1965)  and  Chase 
and  Simon  (1973a,  b),  they  examined  how  ATCSs  recalled  information  over  time.  With  the 
hypothesis  that  ATCSs  store  items  with  related  information  in  chunks,  short  bursts  of  recall 
activity  would  indicate  chunking  because  items  in  the  same  chunk  would  cue  one  another. 

Longer  pauses  between  recalled  items  would  suggest  that  the  previously  recalled  item  did  not 
serve  as  a  cue  for  the  following  item,  and  memory  cues  were  available  from  elsewhere 
(Gronlund  &  Shiffrin,  1986;  Ratcliff  &  McCoon,  1978). 

Means  et  al.  (1988)  conducted  similar  research.  They  too  did  not  find  much  evidence  for 
chunking  because  ATCSs  had  very  few  chunks  that  contained  very  few  aircraft.  However,  both 
Means  et  al.  and  Gronlund  et  al.  (1996)  asked  ATCSs  to  recall  the  airspace  and  aircraft  by 
writing  the  information  on  a  piece  of  paper.  Whereas  this  method  provided  some  data  about 
what  information  is  most  important,  neither  study  was  able  to  fully  support  the  chunking 
hypothesis.  Means  et  al.  asked  the  participants  to  circle  aircraft  that  they  thought  belonged  to  a 
group.  They  based  their  measure  of  chunk  size  solely  on  the  participants  subjective  perception 
of  what  a  chunk  was.  Gronlund  et  al.  used  a  timing  method  similar  to  Chase  &  Simon  (1973a,  b) 
and  failed  to  adequately  measure  chunk  size.  In  the  Gronlund  et  al.  study,  it  took  too  long  for  the 
participants  to  recall  and  write  the  contents  of  their  memory  on  the  paper  map.  The  long  recall 
times  may  have  resulted  in  a  very  insensitive  measure  of  boundaries  between  chunks,  if  such 
chunks  existed. 

3.6.2  Results 

The  analysis  of  the  recall  data  consisted  of  a  2  X  2  (load  X  involvement)  repeated-measures 
ANOVA.  The  tables  in  Appendix  M  detail  these  analyses. 

Both  load  and  involvement  [F(l,  12)  =  24.77  and  5.93  respectively,  both  atp  <  .05,  Table  M-1] 
affected  the  participants  ability  to  recall  aircraft  at  the  end  of  each  scenario.  There  was  no 
significant  load  X  involvement  interaction  (Figure  30).  The  participants  correctiy  recalled  a 
greater  proportion  of  aircraft  under  A  conditions  than  under  M  conditions.  Proportion-correct 
recall  was  dso  greater  under  LL  than  under  HL. 
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Figure  30.  Percent  correct  recall  by  load  and  involvement. 


3.6.3  Discussion 

The  fact  that  the  participants  recalled  more  after  being  actively  involved  in  a  scenario  suggests  a 
deeper  level  of  processing.  This  result  concurs  with  data  collected  using  SPAM  (see  Section 
3.3).  When  the  participants  monitored  the  scenario,  they  may  not  have  been  as  motivated  to 
develop  complex  plans  of  traffic  flow.  The  participants  knew  that  even  if  they  did  devise  a  plan 
to  control  the  air  traffic  in  the  sector,  the  pilots  would  not  carry  out  their  plan  except  by  chance. 
Therefore,  active  involvement  helped  the  participants  to  remember  additional  information  that 
they  did  not  remember  under  M  conditions. 

3.7  Post-Scenario  Questionnaire 


Background 


The  ATCS  responses  to  the  PSQs  provided  information  about  several  aspects  of  ATC  during  a 
particular  simulation  scenario. 


3.7.2  Results 


The  PSQ  was  an  important  source  of  data  that  enabled  the  participants  to  provide  their  opinions 
about  each  experimental  condition.  The  tables  in  Appendix  N  detail  the  results  of  these  analyses. 
We  divided  the  12  items  of  the  PSQ  into  6  groups  for  analysis;  Realism  (Items  1  and  2), 
Workload  (Items  6  and  12),  Interference  (Items  3  and  4),  SA  (Items  8, 9, 10,  and  1 1),  Participant 
Performance  (Item  7),  and  Simulation-Pilot  Performance  (Item  5).  We  analyzed  the  Realism, 
Workload,  Interference,  and  S  A  groups  separately.  Furthermore,  we  analyzed  the  Participant 
Performance  and  Simulation-Pilot  Performance  groups  only  for  a  main  effect  of  load  within  the 
A  condition.  We  did  this  because  neither  participant  nor  simulation-pilot  performance  was 
relevant  during  M  conditions. 
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3.7.2. 1  Realism 


We  conducted  a  2  X  2  (load  X  involvement)  within-subjects  MANOVA  on  Items  1  and  2  of  the 
PSQ,  the  Realism  group.  For  Item  1,  the  participant  described  the  realism  of  the  scenario.  On 
Item  2,  the  participant  rated  how  representative  the  scenario  was  of  a  typical  workday.  The  main 
effects  of  load  and  involvement  were  not  significant  nor  was  the  load  X  involvement  interaction 
(Table  N-1).  The  participants  rated  conditions  in  which  they  actively  controlled  traffic  as  not 
significantly  different  but  more  realistic  and  more  representative  of  a  typical  workday  than  M 
conditions. 

Because  the  multivariate  analysis  did  not  reveal  any  significant  effects,  no  univariate  analysis 
was  necessary.  To  explore  trends  in  the  data,  we  conducted  ANOVAs  on  the  individual  items 
and  looked  for  effects  that  would  be  significant  at  a  more  liberal  alpha  level  of  p  <  .05.  Tables 
N-2  and  N-3  present  the  results  of  the  ANOVAs  on  the  questions  related  to  realism  and 
representativeness.  ATCSs  rated  the  A  control  scenarios  as  more  realistic  than  M  scenarios 
(Figure  31).  There  was  no  difference  in  realism  due  to  a  change  in  load.  ATCSs  perceived  A 
scenarios  to  be  more  representative  of  a  day  at  work  than  the  M  scenarios  (Figure  32).  There 
was  no  effect  of  load  on  the  perceived  representativeness. 

3.7.2.2  Workload 

We  conducted  a  2  X  2  (load  X  involvement)  within-subjects  MANOVA  on  Items  6  and  12,  the 
Workload  group,  of  the  PSQ.  On  Item  6,  the  participants  described  how  hard  they  worked 
during  the  scenario.  For  Item  12,  the  participants  described  the  difficulty  of  the  scenario.  We 
found  a  significant  load  X  involvement  interaction  [F(2, 14)  =  9.30,  p  <  .05,  Table  N-19], 
therefore  we  conducted  simple-effects  MANOVAs  for  the  independent  variables  manipulated 
within  load  (A  vs.  M)  and  involvement  (HL  vs.  LL).  Load  demonstrated  a  significant  effect 
within  A  conditions  [F(2, 14)  =  36.01,  p  <  .05],  but  there  was  no  significant  effect  of  load  within 
M  conditions  (Table  N-3).  The  participants  rated  the  HL  scenario  as  more  difficult  than  the  LL 
scenario  when  they  were  actively  controlling  traffic.  Conversely,  load  during  M  conditions  did 
not  significantly  affect  the  participant  ratings  of  scenario  difficulty.  We  expected  this  result 
because  the  participants  did  not  have  to  make  control  decisions  during  M  conditions.  They  made 
few  keyboard  and  QAK  entries,  and  communications  occurred  only  when  coordinating  with 
adjacent  sectors  and  facilities.  We  also  found  a  significant  effect  of  involvement  within  both  LL 
and  HL  conditions  [F(2, 14)  =  13.81  and  25.99,  p  <.05,  Table  N-4].  The  participants  rated  the  A 
conditions  as  more  difficult  than  M  conditions  regardless  of  load.  Again,  this  is  not  surprising 
because  the  participants  performed  fewer  physical  and  verbal  activities  during  M  conditions. 
Because  the  omnibus  MANOVA  was  significant,  we  conducted  a  separate  ANOVA  on  each  item 
of  the  Workload  group.  An  adjusted  alpha  level  of  a  =  .0253  determined  if  a  result  was 
significant. 
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Figure  31.  Realism  by  load  and  involvement. 
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Figure  32.  Representativeness  by  load  and  involvement. 

For  Item  6,  we  found  a  load  X  involvement  interaction  [F(l,  15)  =  10.65,/?  <  .05,  Table  N-5]. 
Item  6  showed  a  significant  simple  effect  of  load  within  the  M  conditions.  It  showed  a 
significant  simple  effect  for  load  within  A  conditions  [F(l,  15)  =  39.68,  p  <  .05,  Table  N-5].  The 
simple  effects  of  involvement  within  both  LL  and  HL  [F(l,  15)  =  18.77  and  47.59,  p  <  .05,  Table 
N-5]  were  also  significant.  The  participants  made  the  same  ratings  on  average  after  M 
conditions  regardless  of  load.  Compared  to  M  conditions,  the  participants  made  higher  ratings 
after  A  conditions  regardless  of  load  (Figure  33). 
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Figure  33.  Working  hard  by  load  and  involvement. 

For  Item  12,  the  difficulty  of  the  scenario,  we  found  a  significant  load  X  involvement  interaction 
[F(l,  15)  =  5.21,  p  <  .05,  Table  N-6].  Item  12  showed  a  significant  simple  effect  of  load  within 
the  M  conditions.  It  showed  a  significant  simple  effect  for  load  within  A  conditions  [F(l,  15)  = 
23.82,  p  <  .05,  Table  N-6].  The  simple  effects  of  involvement  within  both  LL  and  HL  conditions 
[F(l,  15)  =  1 1.04  and  47.59,  p  <  .05,  Table  N-6]  were  also  significant.  The  participants  made  the 
same  ratings  on  average  after  the  M  conditions  regardless  of  load.  Compared  to  the  M 
conditions,  the  participants  made  higher  ratings  after  the  A  conditions  regardless  of  load  (Figure 
34). 
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Figure  34.  Difficulty  by  load  and  involvement. 
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3.1.23  Interference 


Before  conducting  a  formal  analysis  on  the  interference  of  the  ATWTT  device  and  the 
oculometer,  we  emphasized  that  the  rating  of  the  level  of  interference  of  both  these  devices  was 
very  low  (on  average,  rated  below  4  on  a  10-point  scale).  To  test  if  the  participants  perceived 
any  interference  from  either  device,  a  2  (load)  X  2  (involvement)  within-subjects  MANOVA  was 
conducted  on  Items  3  and  4,  the  Interference  group,  of  the  PSQ.  On  Item  3,  the  participants 
described  how  much  the  ATWIT  device  interfered  with  controlling  traffic.  For  Item  4,  the 
participants  described  how  much  the  oculometer  interfered  with  controlling  traffic.  Significant 
effects  were  found  for  load  and  involvement  [F(2, 14)  =  7.98  and  7.47,  respectively,  both  at  p  < 
.05,  Table  N-7].  The  load  X  involvement  interaction  was  not  significant.  The  participants 
reported  that  there  was  more  interference  from  the  ATWTT  and  the  oculometer  during  HL 
conditions  and  during  A  conditions.  Because  the  omnibus  MANOVA  was  significant,  we 
conducted  a  separate  ANOVA  on  each  item  of  the  Workload  group.  An  adjusted  alpha  level  of 
a  =  .0253  determined  if  a  result  was  significant. 

A  2  X  2  (load  X  involvement)  repeated-measures  ANOVA  was  conducted  on  Item  3  of  the  PSQ. 
We  found  significant  effects  of  both  load  and  involvement  [F(l,  15)  =  17.01  and  12.42, 
respectively,  both  at  p  <  .05,  Table  N-8].  The  load  X  involvement  interaction  was  not 
significant.  The  ATWTT  device  interfered  more  with  controlling  traffic  under  A  conditions  than 
under  M  conditions.  An  increase  in  load  increased  the  interference  of  the  ATWTT  device  and 
more  so  under  A  control  (Figure  35). 
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Figure  35.  ATWTT  interference  by  load  and  involvement. 

3.7.2.4  Situation  Awareness 

We  conducted  a  2  X  2  (load  X  involvement)  within-subjects  MANOVA  on  Items  8,  9, 10,  and 
1 1,  the  SA  group  of  the  PSQ.  On  Item  8,  the  participants  described  their  overall  SA  during  this 
scenario.  For  Item  9,  the  participants  described  their  SA  for  current  aircraft  location.  On  Item 
10,  the  participants  described  their  S  A  for  projected  aircraft  locations.  The  participants  also 
rated  their  SA  for  potential  violations.  Item  11.  We  found  significant  effects  for  the  load  X 
involvement  interaction  [F(4, 12)  =  4.38,  p  <  .05,  Table  N-9]. 
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Because  of  the  significant  load  X  involvement  interaction,  we  conducted  simple-effects 
MANOVAs.  There  was  no  significant  effect  of  load  during  M  conditions.  Load  did  have  a 
significant  effect  during  A  conditions  [F(2, 14)  =  8.37,  p  <  .05,  Table  N-9].  There  was  no 
significant  effect  of  involvement  during  LL  conditions,  but  there  was  a  significant  effect  of 
involvement  during  HL  conditions  [F(2, 14)  =  4.92,  p  <  .05,  Table  N-9].  Because  of  the 
significant  omnibus  MANOVA,  we  conducted  separate  2X2  (load  X  involvement)  ANOVAs 
for  each  of  the  four  items. 

Item  8,  which  asked  about  overall  SA,  yielded  no  significant  results  (Table  N-10).  There  was  a 
trend  visible  for  the  interaction  between  die  effects  of  load  and  involvement  (Figure  36). 


Figure  36.  Overall  S  A  by  load  and  involvement. 

Item  9,  SA  for  current  aircraft  locations,  showed  a  significant  effect  of  load  [F(l,  15)  =  22.70,  p 
<  .05,  Table  N-1 1].  The  ATCSs  rated  the  perceived  SA  for  current  aircraft  locations  higher 
under  LL  than  under  HL.  Although  the  interaction  between  load  and  involvement  did  not  reach 
significance,  there  is  a  trend  visible  as  displayed  in  Figure  37.  The  perceived  heightened 
awareness  for  current  aircraft  positions  under  LL,  A  conditions,  is  responsible  for  the  main  effect 
of  load.  Under  HL,  a  change  in  involvement  does  not  alter  the  perceived  S  A  for  current  aircraft 
positions. 


♦  Monitoring  —■—Active  Control 
Figure  37.  S  A  for  current  aircraft  position  by  load  and  involvement. 
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Item  10,  SA  for  projected  aircraft  locations,  showed  a  significant  effect  of  load  [F(l ,  15)  =  8.72, 
p  <  .05,  Table  N-12].  The  ATCSs  felt  that  they  were  less  aware  of  future  aircraft  positions  under 
HL  than  they  were  under  LL  conditions  (Figure  38). 
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Figure  38.  S  A  for  projected  aircraft  positions  by  load  and  involvement. 

Item  1 1,  SA  for  potential  violations,  showed  a  significant  effect  of  load  [F(l,  15)  =  13.25, 

p  <  .05,  Table  N-13].  Under  HL,  ATCSs  felt  they  had  lower  SA  for  potential  violations  (Figure 

39). 
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Figure  39.  S  A  for  potential  violations  by  load  and  involvement. 

3.7 .2.5  Participant  Performance 

On  Item  7,  the  participants  described  how  well  they  controlled  traffic  during  the  scenario. 
Because  this  question  only  applied  to  the  A  condition,  we  conducted  a  1-way  ANOVA  to  assess 
the  potential  effect  of  load  on  responses  to  this  item.  We  found  a  main  effect  of  load  [F(l ,  15)  = 
13.50,  p  <  .05,  Table  N-14]  indicating  that  the  participants  felt  they  performed  better  under  LL 
conditions  (Table  N-25). 
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3.12.6  Simulation  Pilot  Performance 


The  participants  rated  simulation  pilot  performance  on  Item  5.  Because  the  simulation  pilots 
were  only  present  in  the  A  conditions,  we  performed  a  1-way  ANOVA  to  determine  if  the 
participants  rated  the  simulation  pilots  differently  across  load.  The  lack  of  any  significant 
difference  suggested  that  the  simulation  pilots  performed  equally  well  across  the  A  conditions. 

3.7.3  Discussion 

The  ATCSs  rated  the  A  scenarios  as  more  realistic  than  the  M  scenarios.  The  participants  may 
have  given  slightly  higher  ratings  to  the  A  condition  because  they  typically  control  air  traffic  in 
an  active  manner  and  seldom,  if  ever,  serve  only  as  a  monitor'. 

The  ATCSs  indicated  that  the  A  scenarios  were  more  difficult  than  the  M  scenarios,  although  the 
effect  of  involvement  did  interact  with  load.  ATCSs  rated  the  HL  scenarios  to  be  more  difficult 
than  LL  scenarios. 

The  participants  did  not  think  that  the  oculometer  was  more  intrasive  in  one  condition  than 
another.  However,  some  participants  did  report  that  it  was  easier  to  forget  about  the  oculometer 
when  they  were  actively  engaged  in  the  situation  at  hand.  We  can  contrast  the  oculometer  with 
the  ATWrr  device  because  the  ATWIT  device  requires  physical  activity  and  decision  making 
from  the  participant  where  the  oculometer  does  not.  Therefore,  there  is  no  reason  for  the 
oculometer  to  interfere  differently  depending  on  experimental  conditions.  The  statistical  results 
support  such  a  h5q)othesis. 

The  analysis  of  the  individual  S  A  items  on  the  PSQ  indicated  that  the  participants  thought  that 
the  various  conditions  did  not  affect  their  overall  S  A.  However,  the  participants  felt  that  their 
SA  for  current  and  projected  aircraft  location  and  S  A  for  potential  violations  were  better  under 
LL  conditions.  The  participants  did  not  perceive  a  difference  in  their  SA  between  A  and  M 
conditions.  The  absolute  mean  ratings  suggest  that  the  participants  perception  of  their  SA  and 
their  measured  level  of  SA  may  not  necessarily  agree.  Both  SPAM  and  Recall  tests  showed 
lower  S A  for  the  M  condition,  whereas  perceived  SA  did  not  significantly  change. 

The  expected  perceived  effect  of  involvement  on  the  perceived  SA  was  not  present.  The  ATCSs 
rated  the  SA  for  potential  violation  in  general  better  than  their  SA  for  current  or  projected  aircraft 
positions.  This  finding  corresponds  well  with  the  notion  that  the  ATCS  mostly  store  gist  type  or 
relative  information.  To  be  aware  of  aircraft-specific  positional  information  is  more  difficult  in 
this  case  than  of  potential  violations  (information  about  relations  between  aircraft). 

4.  General  Discussion 

The  current  experiment  investigated  the  effect  of  changing  the  level  of  involvement  on  the  ATCS 
participants.  The  ATCSs  may  move  from  the  active  situation  of  the  current  NAS  to  an 
environment  where  involvement  will  be  more  like  a  monitor  than  a  controller.  This  study 


'  En  route  ATCSs  may  serve  as  a  monitor  during  OTS  training  or  during  recertification. 
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exposed  the  participants  to  the  two  ends  of  the  spectrum  of  involvement.  On  one  end,  the 
ATCSs  carried  out  business  as  usual.  The  ATCS  was  in  control  and  pilots  followed  control 
instructions.  In  the  other  situation,  pilots  maneuvered  their  aircraft  without  control  instructions, 
and  there  was  no  pilot-ATCS  communication. 

To  investigate  the  effect  of  the  change  of  involvement  of  the  controller,  we  employed  five  data 
sets  (rating  form  and  RTOP  results  were  only  applicable  under  A  conditions).  We  have  provided 
brief  discussions  of  the  results  for  each  data  set  separately.  Here,  we  will  provide  insight  into 
how  the  removal  of  control  affected  ATCS  behavior  and  performance  in  general.  We  will  focus 
on  how  involvement  affected  the  ATCS  behavior  and  performance.  The  specific  discussions  on 
each  data  set  address  the  effects  of  load  and  time. 

4.1  Workload 

Perceived  workload  was  higher  under  A  conditions.  Under  M  conditions,  the  estimated 
workload  was  constant  over  time.  Under  A,  on  the  other  hand,  the  estimated  workload  slowly 
increased  over  time.  However,  lower  workload  may  not  necessarily  be  a  desirable  goal, 
depending  on  other  effects.  Overall,  ratings  of  workload  were  low  to  moderate.  This  is  not 
unusual  in  a  population  like  ATCSs,  who  have  a  great  deal  of  experience.  It  takes  a  great  deal  to 
move  them  beyond  a  moderate  workload  rating. 

4.2  Situation  Awareness 

The  SPAM  asks  the  participants  questions  about  present  and  future  situations.  The  time  to 
answer  a  question  is  an  indicator  of  how  quick  a  participant  can  access  relevant  information.  A 
change  in  the  level  of  involvement  did  not  affect  answers  to  questions  about  future  situations. 
However,  ATCS  involvement  did  affect  the  ability  to  answer  questions  related  to  the  present 
situation.  Under  A  conditions,  the  time  to  answer  the  queries  about  the  present  situation  was 
equal  for  LL  and  HL  scenarios.  Under  M  conditions,  the  increase  in  load  almost  doubled  the 
time  to  answer  the  questions.  The  SPAM  does  not  probe  memory.  All  information  necessary  to 
answer  the  questions  is  available  bn  the  radarscope.  The  fact  that,  under  M  conditions,  the 
ATCS  takes  longer  to  answer  the  SPAM  queries  is  an  indication  that  S  A  suffers  from  reduced 
involvement.  This  is  contrary  to  the  beliefs  of  those  that  suggest  that  a  monitoring  situation  will 
free  cognitive  resources.  Freeing  cognitive  resources  would  allow  the  ATCS  to  direct  more 
resources  to  keep  an  up-to-date  picture  of  the  situation.  The  current  results  are  more  in  line  with 
earlier  findings  that  working  memory  for  something  that  you  have  done  yourself  is  better  than 
something  that  someone  does  for  you. 

The  PSQ  asked  the  ATCSs  about  their  opinion  on  their  SA  for  aircraft  positions  and  potential 
violations.  The  ATCSs  indicated  that,  although  an  increase  in  load  reduced  their  SA  for  aircraft 
positions  and  potential  violations,  the  reduction  of  involvement  did  not  affect  their  S  A.  This  is  in 
sharp  contrast  with  the  findings  from  the  objective  measure  of  SA.  Therefore,  although  the 
ATCS  may  not  be  aware  that  the  SA  is  suffering  when  monitoring  traffic,  the  actual  S A  is  not  as 
good  under  M  conditions  as  it  is  under  A  conditions.  Increased  automation  or  changes  in  the 
NAS  that  will  place  the  ATCS  in  a  monitoring  position  may  give  the  ATCS  a  false  feeling  of 
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having  good  S  A,  whereas  S  A  has  already  diminished.  Counter  measures  to  assist  the  ATCS  in 
maintaining  an  accurate  S A  may  be  necessary  when  changes  in  the  NAS  require  the  ATCS  to 
become  a  monitor. 

4.3  Eve  Movements 

The  general  characteristics  of  eye  movements  did  not  change  by  load  or  involvement.  The  effect 
of  time,  on  the  other  hand,  affected  the  number  and  the  duration  of  fixations.  During  the  first  5 
minutes  of  the  simulations,  the  ATCSs  scanned  for  information  with  more  and  shorter  fixations 
than  during  the  rest  of  the  simulation  time.  A  possible  explanation  is  that  the  ATCSs  received  a 
relief  briefing  at  the  start  of  the  simulation.  The  ATCSs,  therefore,  merely  verified  the 
correctness  of  the  information  in  the  beginning  of  the  simulation.  Other  studies  have  shown  that 
several  categories  of  fixations  exist.  Carmody,  Nodine,  and  Kundel  (1981)  distinguish  surveying 
(short  duration)  fixations  and  evaluating  (long  duration)  fixations  in  radiologists  scanning  X- 
rays.  The  need  to  acquire  all  information  related  to  the  current  situation  only  becomes  critical 
once  the  state  of  the  airspace  has  changed  considerably.  When  the  ATCS  moves  into  a  state  of 
information  acquisition  and  monitoring  instead  of  verification  and  monitoring,  the  duration  of 
fixations  increases,  and  the  number  of  fixations  decreases. 

Fixation  area  tended  to  increase  over  time.  The  first  5  minutes  showed  more  stable  fixations 
than  subsequent  5-minute  intervals.  There  are  several  possible  explanations  for  this  finding. 
First,  during  the  relief  briefing,  the  ATCS  receives  specific  information  about  particular  aircraft 
and  may  focus  on  these  aircraft  while  digesting  the  information.  Fixations  will  not  fall  within 
clusters  of  aircraft,  and  small  adjustments  may  not  be  necessary.  After  the  ATCS  takes  over 
control,  these  changes  and  fixations  become  less  stable.  An  alternative  explanation  is  that  the 
visual  system  shows  signs  of  fatigue.  Although  research  has  shown  that  eye  movements  can 
continue  for  long  periods  without  showing  signs  of  fatigue,  the  number  of  glissades,  or  slipping 
into  or  out  of  a  fixation,  increase  with  fatigue.  Our  algorithm  to  calculate  fixation  onset  and  area 
may  have  captured  glissades  as  well,  thereby  increasing  the  average  fixation  area  with  an 
increase  in  the  number  of  glissades. 

The  effects  of  the  manipulation  of  load  and  involvement  only  became  apparent  during  analyses 
of  fixation  characteristics  broken  down  by  scene  planes  and  radarscope  objects.  Most  fixations 
landed  on  the  radarscope,  followed  by  the  flight  strip  bay  and  the  QAK/CRD.  Fixations  on  the 
map  and  the  QAK/CRD  were  shorter  under  M  than  under  A  conditions.  The  ATCSs  had  little 
need  for  both  QAK/CRD  and  the  map  under  M  conditions  and  spent  less  time  retrieving 
information  from  these  displays.  During  A  conditions,  the  ATCSs  used  the  QAK/CRD  as  both  a 
data  entry  and  data  display  tool  when  assigning  altitudes,  and  so  on.  Under  M  conditions,  the 
QAK/CRD  was  merely  there,  and  the  ATCSs  only  looked  at  it  briefly  to  verify  data  entry  for 
data  block  movement,  not  for  control  actions.  Therefore,  monitoring  does  change  how 
controllers  use  displays. 

The  ATCSs  focused  most  of  their  fixations  on  the  radar  returns  and  the  datablocks.  In  addition 
to  the  increased  number  of  fixations  on  these  two  objects,  these  fixations  were  considerably 
longer  than  fixations  on  any  of  the  other  objects  or  scene  planes. 
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To  explore  the  structure  or  predictability  in  the  visual  scan  of  the  ATCSs,  we  developed  four 
indices  based  on  “conditional  information”  (Ellis,  1986).  These  indices  investigated  the 
distribution  of  radarscope  fixations  across  the  radarscope  (location  and  distance  from  the 
radarscope  center),  across  objects,  and  broken  down  by  inter-fixation  distances.  Although  the 
radarscope  location-based  index  showed  the  highest  level  of  structure,  it  did  not  change 
significantly  with  a  change  in  our  conditions.  The  fact  that  this  index  showed  higher  levels  of 
structure  may  stem  from  the  existence  of  structure  in  the  airspace.  One  would  expect  that  the 
values  for  this  index  would  decrease  when  the  structure  in  traffic  flow  is  less  apparent,  as  would 
be  the  case  in  Free  Flight.  The  index  that  focused  on  structure  in  the  visual  scan  based  on 
distance  from  the  center  of  the  radarscope  did  not  reveal  an  effect  of  load  or  involvement 
manipulation  either.  In  the  current  en  route  experiment,  the  ATCSs  did  not  have  a  “sink”  like 
the  main  airport  often  encountered  in  TRACON  environments.  One  would  expect  more  structure 
in  the  ATCSs  visual  scan  based  on  this  index  due  to  the  structure  of  the  TRACON  airspace.  It  is 
more  likely  that  a  fixation  on  a  part  of  the  TRACON  high  traffic  area  will  follow  by  a  fixation  on 
another  high  traffic  area. 

Load  affected  the  structure  in  the  visual  scan  when  based  on  target  objects.  Although  the 
structure  was  low,  an  increase  in  load  reduced  the  predictability  of  the  visual  scan.  The 
reduction  in  scanning  structure  due  to  active  involvement  was  only  apparent  under  HL.  The 
ATCSs  seemed  to  scan  the  radarscope  in  a  more  random  fashion  when  the  complexity  increased 
and  they  actively  controlled  traffic.  The  way  the  ATCSs  distribute  their  attention  across 
radarscope  objects  does  not  alter  when  their  task  is  to  monitor  traffic.  Therefore,  the  ATCSs  are 
less  likely  to  adapt  their  scanning  behavior  with  a  change  in  the  traffic  situation. 

Our  final  index  investigated  how  likely  it  was  that  fixations  with  particular  inter-fixation 
distances  follow  one  another  in  a  fixed  pattern.  The  results  show  that  this  is  more  likely  to 
happen  under  HL.  This  does  not  necessarily  mean  that  the  ATCSs  are  more  likely  to  suffer  from 
tunnel  vision.  It  could  mean  that  it  is  more  likely  that  a  fixation  with  a  short  inter  fixation 
distance  often  follows  a  fixation  with  a  long  inter-fixation  distance.  More  detailed  analyses  of 
the  transition  probability  matrix  that  focuses  on  the  likelihood  that  fixations  with  short  inter 
fixation  distances  follow  one  another  would  allow  the  determination  of  the  occurrence  of  tunnel 
vision. 

When  we  removed  active  control  from  the  ATCS,  we  expected  a  change  in  eye  movement 
characteristics.  Under  monitoring  conditions,  the  expected  need  for  information  is  less. 
Consequently,  one  would  expect  that  the  fixation  duration  and  frequency  would  decrease.  When 
the  ATCSs  are  no  longer  actively  changing  the  state  of  the  aircraft  in  the  airspace,  the  need  to 
evaluate  the  current  state  and  the  outcome  of  actions  no  longer  exists.  The  need  for  evaluation- 
type  fixations  of  longer  duration  would  decrease.  With  the  loss  of  the  bigger  picture,  the  ATCS 
would  be  less  likely  to  look  for  information  in  an  open-loop  fashion  guided  by  higher  level  goals. 
This  would  result  in  a  scanning  pattern  that  more  relies  on  local  feedback  of  the  events  on  the 
radarscope.  The  local  feedback  in  the  visual  scanning  pattern  ought  to  lead  to  a  larger  statistical 
dependency  expressed  in  more  structure  or  higher  values  of  the  conditional  information  indices 
in  monitoring.  The  visual  scan  showed  less  structure  under  active  control  than  under  monitoring 
conditions.  Scanning  for  information  in  the  open-loop  fashion  by  definition  means  less  structure. 
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We  suspect  that  during  monitoring,  ATCSs  establish  a  stimulus-driven  scan  that  is  more 
stractured.  Interestingly  enough,  the  manipulation  of  the  level  of  involvement  did  not  change  the 
eye  movement  characteristics. 

The  literature  explains  these  findings.  It  takes  a  considerable  amount  of  practice  to  teach  the 
visual  system  something  until  it  becomes  automatic.  Automaticity  in  visual  information 
processing  implies  rapid,  parallel  processing.  Once  a  person  learns  a  task,  until  automaticity 
occurs,  the  task  at  hand  requires  very  little  cognitive  resources.  This  type  of  task  performance  is 
quite  common  among  domain  experts.  The  characteristics  of  the  structure  in  the  visual  scan  after 
automaticity  sets  in,  contrary  to  the  training  process  itself,  are  visible  early  on.  Within  30 
minutes  for  simple  stimuli,  the  parameters  that  establish  the  visual  scanning  pattern  will  emerge 
(Moray,  1986).  In  the  current  experiment,  we  removed  the  ATCS  active  involvement  in  the  task 
at  hand.  The  presentation  format  of  the  display  elements,  however,  remained  the  same.  This 
resulted  in  information  acquisition  behavior  that  did  not  change  (e.g.,  the  eye  fixation  durations 
and  frequencies  remained  relatively  constant).  The  structure  in  the  visual  scan,  on  the  other 
hand,  did  show  effects  of  the  change  in  involvement  as  indicated  by  the  changes  in  the 
conditional  information  indices. 

Willems  et  al.  (in  press)  modeled  the  home  sector  of  a  group  of  TRACON  ATCSs.  The  ATCSs 
had  worked  their  airspace  for  several  years  and  were  quite  familiar  with  the  traffic  patterns.  This 
familiarity  may  have  led  the  ATCSs  to  develop  efficient  visual  information  acquisition  processes 
that  have  increased  the  visual  lobe  size  (the  area  of  the  visual  field  that  an  ATCS  can  efficiently 
use  to  retrieve  information).  Although  fixation  durations  are  longer,  ATCSs  process  more 
information  in  the  periphery.  The  increase  in  visual  lobe  size  makes  it  easier  to  combine 
information  about  several  aircraft.  The  more  advanced  integration  of  information  about  several 
aircraft  in  a  single-eye  fixation  would  result  in  more  efficient  scanning  patterns.  The  ARTCC 
ATCSs  participating  in  the  current  study  worked  an  unfamiliar  airspace  and  did  not  have  the 
advantage  of  working  that  airspace  for  many  years.  Consequently,  the  peripheral  processing  of 
information  could  not  take  advantage  of  background  knowledge  learned  from  experience 
resulting  in  a  smaller  functional  field  of  view  and  less  information  to  absorb  at  a  time.  The 
reduction  in  information-per-fixation,  in  turn,  would  lead  to  shorter  fixation  durations  and  more 
fixations. 

In  the  TRACON  environment,  the  ATCSs  did  not  have  the  option  to  extend  the  leader  lines  that 
connect  the  radar  return  and  the  data  block.  The  data  block  and  the  radar  return  were  in  close 
proximity  of  one  another.  For  the  ATCSs  that  are  very  familiar  with  the  aircraft  representation, 
this  allows  them  to  absorb  all  information  relevant  for  a  given  aircraft  in  a  single  fixation.  The 
fact  that  this  single  fixation  now  can  pick  up  more  information  will  necessitate  a  longer  duration 
for  information  retrieval.  In  the  ARTCC  environment,  the  ATCSs  seemed  to  keep  the  data 
blocks  at  a  larger  distance  from  the  radar  return.  To  foveate  all  information  for  a  single  aircraft, 
the  ATCSs  may  require  two  fixations  instead  of  one.  Less  information  retrieval  takes  place  for 
each  of  these  fixations,  leading  to  shorter  fixations. 

An  ATCS  in  the  TRACON  airspace  faces  a  lack  of  structure  compared  to  the  structured  airspace 
of  the  ARTCC  environment.  The  ARTCC  ATCS  can  fall  back  on  a  large  number  of 
assumptions  based  on  where  an  aircraft  is  within  the  airspace.  The  amount  of  information  that 
the  ATCS  needs  to  retrieve  for  a  given  aircraft  in  the  ARTCC  environment  may  be  less  than  in 
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the  TRACON  environment.  The  reduction  of  the  amount  of  information  that  the  ATCS  retrieves 
by  using  assumptions  stored  in  long  term  memory  will  lead  to  shorter  retrieval  times  and, 
therefore,  shorter  fixation  durations. 

The  ATCSs  have  several  types  of  fixations.  When  reading  general  information,  the  ATCS  will 
perform  just  like  any  other  reader.  The  ATCS  visual  scanning  system,  however,  must  have 
developed  a  level  of  automaticity  that  a  non-ATCS  does  not  have.  The  longer  fixations  on 
aircraft  are  an  indication  of  that.  The  controller  is  picking  up  relevant  information  from  an 
aircraft  representation.  The  ATCS  does  that  faster  than  non-ATCSs.  The  TRACON  data  block 
alone  consists  of  call  signs,  computer  IDs,  altitude,  and  speed  (4  items).  The  radar  return  and 
everything  attached  to  that  consists  of  the  position  symbol,  vector  line,  and  history  trace  (3 
items).  That  could  take  up  to  7  fixations  if  the  ATCS  would  scan  for  information  in  a  sequential 
manner  (no  automaticity  or  parallel  processing).  Just  to  prepare  for  the  next  saccade  takes  about 
75  msecs.  At  least  the  same  amount  of  time  is  needed  for  the  acquisition  of  simple  information 
from  scenes  such  as  photographs.  If  we  omit  the  time  to  process  the  information  to  decide  where 
to  jump  to  next,  the  visual  system  needs  150  msecs  to  get  the  information  and  to  move  on  to  the 
next  spot.  That  times  seven  would  give  us  a  little  over  one  second  to  visit  all  elements  of  the 
aircraft  representation.  With  processing  of  the  information,  the  controller  does  this  in  a  little 
over  600  msecs.  In  addition,  the  ATCS  may  do  that  not  for  just  one  aircraft  but  for  other  aircraft 
that  are  in  the  parafoveal  (an  area  of  between  one  and  three  degrees  of  visual  angle  outside  of  the 
center  of  fixation)  and  near  peripheral  areas  of  the  retina). 

Now,  within  these  longer  fixations  on  aircraft,  one  can  still  distinguish  between  surveying  and 
evaluating  fixations.  Surveying  fixations  are  shorter  and  are  likely  terminated  when  the 
controller  decides  at  the  feature  level  that  this  does  not  contain  relevant  information  (the  state  of 
the  aircraft  is  not  changed  or  the  aircraft  does  not  pose  a  potential  problem).  Those  fixations  are 
probably  less  that  350  msec.  During  evaluating  fixations,  the  controller  is  really  picking  up 
information  far  beyond  the  feature  level.  The  ATCS  looks  at  that  aircraft  for  a  purpose  and 
composes  the  overall  picture  of  the  state  of  that  aircraft.  Those  fixations  are  quite  long,  more  in 
the  order  of  500  msecs  and  over. 

5.  Conclusions 

The  current  experiment  placed  the  ATCSs  in  a  monitoring  situation.  Changes  in  airspace 
management  may  move  the  ATCS  to  a  situation  that  will  fall  somewhere  between  the  current, 
active  control  situation  and  the  simulated  monitoring  situation  of  this  study.  The  results  indicate 
that,  although  perceived  workload  is  less  under  monitoring  conditions,  the  objective  S  A 
measures  show  that  ATCSs  SA  declines  substantially  when  the  ATCS  no  longer  actively 
controls  traffic.  The  fact  that  the  ATCS  may  not  be  aware  of  the  reduction  in  SA  suggests  that 
system  designers  must  seriously  consider  how  they  are  going  to  keep  controllers  involved. 
Although  our  experiment  may  have  been  brief,  the  visual  scanning  patterns  showed  changes. 
These  small  changes  after  only  a  brief  exposure  to  work  as  a  monitor  may  be  an  indication  of 
changes  in  eye  movement  characteristics  when  the  ATCS  will  work  in  a  monitoring  role  for 
longer  periods.  Changes  in  the  characteristics  of  eye  movements  are  an  indication  of  visual 
information  retrieval  strategies.  The  altered  SA,  in  combination  with  a  change  in  information 
retrieval  strategies,  warrants  careful  examination.  It  implies  a  need  for  training  and  assistance  of 
the  ATCSs  in  situations  where  they  are  no  longer  in  active  control. 
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Acronyms 


A 

Active  Condition 

ADS-B 

Automatic  Dependent  Surveillance-Broadcast 

ANOVA 

Analysis  of  Variance 

ARTCC 

Air  Route  Traffic  Control  Center 

ASL 

Applied  Science  Laboratories 

ATC 

Air  Traffic  Control 

ATwrr 

Air  Traffic  Workload  Input  Technique 

CRD 

Computer  Readout  Device 

DRA 

Data  Reduction  and  Analysis 

FAA 

Federal  Aviation  Administration 

FL 

Flight  Level 

FPL 

Full  Performance  Level 

FPS 

Flight  Progress  Strip 

GPS 

Global  Positioning  System 

HFS 

Human  Factors  Specialist 

HL 

High  Load 

LL 

Low  Load 

LOA 

Letter  of  Agreement 

M 

Monitoring  Condition 

MANOVA 

Multivariate  Analysis  of  Variance 

NRP 

National  Route  Program 

OJT 

On-the-Job  Training 

OTS 

Over-the-Shoulder 

PEQ 

Post-Experimental  Questionnaire 

PVD 

Plan  View  Display 

PSQ 

Post-Scenario  Questionnaire 

QAK 

Quick  Action  Key 

RDHFL 

Research  Development  and  Human  Factors  Laboratory 

RTOP 

Real  Time  Objective  Performance 

SA 

Situation  Awareness 

SAGAT 

Situation  Awareness  Global  Assessment  Technique 

SCRD 

Soft  Computer  Readout  Device 

SME 

Subject  Matter  Expert 

SOP 

Standard  Operating  Procedure 

SPAM 

Situation  Present  Assessment  Technique 

TCAS 

Traffic  Alert  and  Collision  Avoidance  System 

TRACON 

Terminal  Radar  Approach  Control 

VOR 

Very  High  Frequency  Omnidirectional  Range 

WAAS 

Wide  Area  Augmentation  System 
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Appendix  A 


Genera  Center  Standard  Operating  Procedures  and  Letters  of  Agreement 

U.  S.  Department  of  Transportation 
Federal  Aviation  Administration 
GENERA  ARTCC 

SUB.T;  GENERA  CENTER  STANDARD  OPERATING  PROCEDURES  (SOP^ 

1.  PURPOSE:  This  Order  transmits  ZGX  Genera  Center  Standard  Operating  Procedures. 

2.  DISTRIBUTION :  This  Order  is  distributed  to  facility  managers,  staff  offices,  NATCA, 
NAGE,  control  room  personnel,  and  the  facility  library  at  Genera  ARTCC. 

3.  EFFECTIVE  DATE:  July  20, 1995 

4.  TEAM  POSITION  RESPONSIBILITIES:  En  route  sector  team  responsibilities  are  contained 
in  FAA  Order  71 10.65,  chapter  2,  section  10,  paragraphs  2-130. 

a.  Flight  Data  Position  shall: 

(1)  Prepare  strips  displaying  red  routings  or  red  coordination  symbols. 

(2)  Prepare  strips  for  aircraft  that  will  proceed  to  special  use  airspace  for  which  an 
operational  count  is  authorized. 

(3)  Place  strips  above  the  sector  suspense/active  bayheader  and  sequence  strips  by 
time,  when  appropriate,  with  the  earliest  time  at  the  bottom  of  the  bay. 

(4)  Forward  a  copy  of  the  Traffic  Management  message  to  the  ASIC/CIC.  The 
ASIC/CIC  shall  be  responsible  for  hand  carrying  or  verbally  notifying  the 
appropriate  sector(s). 

b.  Radar  Position  shall: 

(1)  Recognize  sector  saturation  and  employ  procedures  to  prevent  or  alleviate  this 
problem. 

c.  Transfer  of  Radar  Identification. 

(1)  Data  blocks  displaying  verified  MODE  C  information  may  be  used  to 
accomplish  altitude  coordination.  Assigned  altitude  shall  be  reflected  in  the  data 
block  either  as  a  temporary  altitude  or  as  a  final  altitude. 
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(2)  Resolve  all  potential  conflicts  prior  to  dropping  full  data  blocks.  Full  data 
blocks  shall  be  displayed  on  all  aircraft  within  the  confines  of  your  airspace. 

5.  AUTONOMOUS  OPERATIONS  IN  FACSFACGAT  WARNING  AREAS: 

a.  Information. 

(1)  Warning  areas  in  Genera  Center  area  are  established  with  a  designated  using 
agency  and  an  ATC  point  of  contact. 

(2)  The  authorize  representatives  for  activation  and  coordination  of  the  subject 
warning  areas  are  as  follows: 

(a)  W500  (Hotwater)... . . . Plumber  Control 

(3)  Genera  Center  controllers  should  allow  entry  to  W500  at  point  Boill  (depicted 
on  Annex)  at  FL280  and  departure  from  point  Finis  at  FL290,  unless  otherwise 
coordinated. 

6.  GENERA  CENTER  SPECIAL  USE  AIRSPACE  INTRUSION/SPILL  OUT  PROCEDURES: 

(1)  FAA  or  pilot  requests  to  transit  special  use  airspace  to  avoid  weather  do  not  have 
priority  over  military  operations  being  conducted  in  special  use  airspace.  The  decision  to 
release  special  use  airspace  to  the  FAA  rests  solely  with  the  using  agency. 

(2)  Whiskey  Alert  Procedures. 

(a)  The  phrase  “Whiskey  Alert”  shall  be  used  when  spill  in  or  spill  out  from  a 
MOA,  ATCAA,  restricted  area,  or  warning  area  has  not  been  coordinated  or 
approved  in  advance  and  the  spill  in/spill  out  is  imminent. 
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1.  SECTOR  10 . ALPHA  fflGH 

This  sector  shall  include  all  airspace  from  FL240  and  above, 
a.  Standard  Operating  Procedures 

(1)  Aircraft  filed  into  the  Genera  High  Sector: 

(a)  Landing  UTN  shall  be  cleared  NWTJ74.UPPER.UPPER1  or 
NTH.J75.J74.UPPER.UPPER1  at  or  below  FL370. 

(b)  Landing  DTN  shall  be  cleared  SWT.J64.LOWER.LOWER1  or 
NTH.J75.LOWER.LOWER1  at  or  below  FL370 

(c)  Eastbound  overflight  traffic  will  not  be  cleared  via  J70  eastbound. 

(d)  Southbound  overflight  traffic  form  NTH  will  be  established  on  J75  or 
direct  STH  at  or  above  FL330. 

(e)  Aircraft  operating  between  the  Alpha  High  sector  and  the  Genera  High 
sector  will  be  at  even  altitudes  south  and  westbound;  odd  altitudes  north 
and  eastbound. 

2.  SECTOR  11 . BRAVO  HIGH 

This  sector  shall  include  all  airspace  from  FL240  and  above, 
a.  Standard  Operating  Procedures 

(1)  Aircraft  filed  into  the  Genera  High  Sector: 

(a)  Departing  UTN  shall  be  cleared  UTN.UPTWN  1  .MIDLE.  J70  with 
release  for  climb. 

(b)  Departing  DTN  shall  be  cleared  DTN.DNTWNl. MIDLE. J70  with 
release  for  climb. 

(c)  Overflight  traffic  will  be  established  direct  MIDLE  at  a  point  20NM 
east  of  MIDLE. 

(2)  Aircraft  operating  between  the  Bravo  High  Sector  and  Genera  High  Sector 
will  be  at  even  altitudes  south  and  westbound;  odd  altitudes  north  and  eastbound. 
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3.  GENERA  fflGH  SECTOR 


This  sector  shall  include  all  airspace  from  FL240  and  above,  excluding  that  airspace  delegated  to 
ZCX,  FL270  and  above. 

a.  Standard  Operating  Procedures 

(1)  Aircraft  filed  into  the  Alpha  High  Sector: 

(a)  At  or  above  FL240  may  be  cleared  MIDLE  direct  WST  flight  plan 
route. 

(b)  Genera  Sector  shall  ensure  that  aircraft  filed  over  WST  with  the  same 
destination  will  be  in-trail  of  each  other  regardless  of  altitude. 

(2)  Alpha  High  Sector  shall  deliver  arrivals  to  UTN  and  DTN  at  or  below  FL  370. 

(3)  Aircraft  operating  between  the  Genera  High  Sector  and  the  Alpha  High  Sector 
will  be  at  even  altitudes  south  and  westbound;  odd  altitudes  north  and  eastbound. 

(4)  Aircraft  filed  into  the  Bravo  High  Sector: 

(a)  Landing  UTN  shall  be  cleared  via  the  UPPERl  arrival  to  cross  UPPER 
at  FL250. 

(b)  Landing  DTN  shall  be  cleared  via  the  LOWERl  arrival  to  cross 
LOWER  at  FL240. 

(c)  Eastbound  overflight  traffic  shall  be  established  on  J64  or  J74. 
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Genera  Center  Letter  of  Agreement 
Subject:  Liter-Center  Procedures 

Purpose.  This  agreement  establishes  Inter-Center  procedures  between  Charlie  ARTCC  and 
Genera  ARTCC  and  is  supplementary  to  the  procedures  in  the  Air  Traffic  Control  Handbook. 

Effective  Date.  July  20, 1995. 

Responsibilities.  This  agreement  covers  coordination  procedures,  altitude  assignments,  route 
assignments,  delegation  of  airspace,  and  coordination/notification  procedures  of  special  use 
airspace.  Deviation  from  procedures  outlined  in  this  agreement  made  by  either  facility  may  be 
made  only  after  coordination,  which  completely  defines  responsibility  in  each  case. 

Procedures. 

Route  Assignments. 

Traffic  entering  the  Genera  High  sector  shall  be  established  on  J75  at  or  prior  to  the  common 
Center  boundary,  with  the  following  exception: 

Aircraft  at  FL270  and  above  shall  be  established  on  J75  prior  to  the  ZGX/ZCX  center  boundary 
southbound. 

Altitude  Assignment. 

Aircraft  on  J75  shall  be  cleared  northbound  at  odd  altitudes  and  southbound  at  even  altitudes. 

Aircraft  entering  the  Charlie  High  or  Low  sectors  shall  be  at  an  assigned  altitude  designated  by 
the  hemispheric  altitude  for  direction  of  flight. 

Aircraft  entering  the  Genera  High  or  Low  sectors  shall  be  at  an  assigned  altitude  designated  by 
the  hemispheric  altitude  for  direction  of  flight. 
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Appendix  B 
Genera  Center  Airspace 

Genera  sector  controls  traffic  within  its  boundaries  from  24,000  feet  (flight  level  (FL)  240)  and 
above.  All  airways  within  the  airspace  are  one-way  airways.  Two  airways,  J64  and  J74,  move 
traffic  from  west  to  east.  One  airway,  J70,  moves  traffic  from  east  to  west,  and  one  airway,  J75, 
moves  traffic  from  north  to  south.  Tliere  are  eight  Very  High  Frequency  Omnidirectional  Range 
(VOR)  navigational  beacons  associated  with  Genera  sector:  CTR,  Center;  NTH,  North;  NET, 
Northeast;  SET,  Southeast;  STH,  South;  SWT,  Southwest;  WST,  West;  and  NWT,  Northwest. 
There  are  four  intersections  associated  with  the  airspace:  UPPER,  LOWER,  MIDLE,  and 
BOTTM.  Of  these  VORs  and  intersections,  only  CTR  and  MIDLE  are  within  the  airspace. 

Three  airports  are  of  relevance  to  Genera  sector:  UTN,  Uptown  Airport;  MID,  Midtown  Airport; 
and  DTN,  Downtown  Airport.  Genera  sector  lies  between  three  sectors.  On  the  west  side  lies 
Alpha  sector.  To  the  east  lies  Bravo  sector.  Both  Alpha  and  Bravo  sectors  are  from  the  same 
ARTCC.  To  the  south  lies  one  sector  from  another  ARTCC,  Charlie  Center,  and  an  area  of 
restricted  airspace  called  Hotwater  or  W500.  Below  the  Genera  High  sector  is  Genera  Low, 
which  controls  traffic  from  FL  230  and  below.  Although  the  airspace  map  depicts  an  altitude 
shelf,  the  present  experiment  did  not  use  this  shelf. 

Aircraft  had  standard  arrival  and  departure  procedures.  Aircraft  landing  at  UTN  had  to  cross  the 
UPPER  intersection  at  FL  250.  Aircraft  landing  at  DTN  had  to  cross  the  LOWER  intersection  at 
FL  240.  Genera  sector  did  not  control  aircraft  landing  at  MID.  The  ATCS  responsible  for 
aircraft  in  Genera  sector  had  control  for  climb  (e.g.,  without  coordination  from  adjacent  sectors) 
for  aircraft  departing  from  all  three  airports.  However,  permission  to  turn  aircraft  not  within  the 
confines  of  Genera  sector  required  coordination  with  the  appropriate  sector.  Aircraft  travelling 
to  the  same  destination  airport  required  at  least  5  NM  in-trail  separation,  regardless  of  altitude. 
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Appendix  C 

Observer  Rating  Form,  Instinctions,  and  Rating  Criterion 
OBSERVER  RATING  FORM 

Observer  Code _  Date _ 

Participant: 

Scenario: _ 

INSTRUCTIONS 

This  form  is  designed  to  be  used  by  supervisory  air  trafHc  control  specialists  to  evaluate  the 
effectiveness  of  controllers  working  in  simulation  environments.  SATCSs  will  observe  and 
rate  the  performance  of  controllers  in  several  different  performance  dimensions  using  the 
scale  below  as  a  general-purpose  guide.  Use  the  entire  scale  range  as  much  as  possible. 

You  will  see  a  wide  range  of  controller  performance.  Take  extensive  notes  on  what  you  see. 
Do  not  depend  on  your  memory.  Write  down  your  observations.  Additional  pages  are 
provided  for  your  comments.  Please  indicate  category  number  to  which  you  are  referring. 
You  may  make  preliminary  ratings  during  the  course  of  the  scenario.  However,  wait  until 
the  scenario  is  finished  before  making  your  final  ratings  and  remain  flexible  until  the  end 
when  you  have  had  an  opportunity  to  see  the  entire  available  behavior.  At  aU  times  please 
focus  on  what  you  actuaUy  see  and  hear.  This  includes  what  toe  controller  does  and  what 
you  might  reasonably  infer  from  the  actions  of  the  pilots.  Try  to  avoid  inferring  what  you 
think  may  be  happening.  If  you  do  not  observe  relevant  behavior  or  toe  results  of  that 
behavior,  then  you  may  leave  a  specific  rating  blank.  Also,  please  write  down  any 
comments  that  may  help  improve  this  evaluation  form.  Do  not  write  your  name  on  the 
form  itself.  Your  identity  wUl  remain  anonymous,  as  your  data  will  be  identified  by  an 
observer  code  known  only  to  yourself  and  the  researchers  conducting  this  study.  The 
observations  you  make  do  not  need  to  be  restricted  to  the  performance  areas  covered  in 
this  form  and  may  include  other  areas  that  you  think  are  important. 

Assiimntions!  ATC  is  a  complex  activity  that  contains  both  observable  and  unobservable 
behavior.  There  are  so  many  complex  behaviors  involved  that  no  observational  rating 
form  can  cover  everything.  A  sample  of  the  behaviors  is  the  best  that  can  be  achieved,  and 
a  good  form  focuses  on  those  behaviors  that  controllers  themselves  have  identified  as  the 
most  relevant  in  terms  of  their  overall  performance.  Most  controller  performance  is  at  or 
above  the  minimum  standards  regarding  safety  and  efficiency.  The  goal  of  toe  rating 
system  is  to  differentiate  performance  above  tMs  minimum.  The  lowest  rating  should  be 
assigned  for  meeting  minimum  standards  and  for  anything  below  the  minimum  since  this 
should  be  a  rare  event  It  is  important  for  the  observer/rater  to  feel  comfortable  using  toe 
entire  scale  and  to  understand  that  all  ratings  should  be  based  on  behavior  that  is  actually 
observed. 
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1 

Least 

Effective 

Unconfident,  Indecisive,  Inefficient, 
Disorganized,  Behind  the  power  curve.  Rough, 
Leaves  some  tasks  incomplete.  Makes 
mistakes 

2 

Poor 

May  issue  conflicting  instructions;  Does  not 
plan  completely 

3 

Fair 

Distracted  between  tasks 

Low  Satisfactory  Postpones  routine  actions 


High  Satisfactory  Knows  the  job  fairly  well 


Good  Works  steadily,  Solves  most  problems 


Very  Good  Knows  the  job  thoroughly.  Plans  well 


Very  Good 


Most 

Effective 


Confident,  Decisive,  Efficient,  Organized, 
Ahead  of  the  power  curve.  Smooth,  Completes 
all  necessary  tasks.  Makes  no  mistakes _ 


I  -  MAINTAINING  SAFE  AND  EFFICIENT  TRAFFIC  FLOW 

1.  Maintaining  Separation  and  Resolving  Potential  Conflicts . 1  2  3  4  5  6  7  8 

•  using  control  instructions  that  maintain  appropriate  aircraft  and 

airspace  separation 

•  detecting  and  resolving  impending  conflicts  early 

•  recognizing  the  need  for  speed  restrictions  and  wake  turbulence 

separation 

2.  Sequencing  Arrival,  Departure,  and  En  Route  Aircraft  Efficiently  ..1  2  3  4  5  6  7  8 

•  using  efficient  and  orderly  spacing  techniques 

•  maintaining  safe  arrival  and  departure  intervals  that  minimize 

delays 

3.  Using  Control  Instructions  Effectively/Efficiently . 12345  678 

•  providing  accurate  navigational  assistance  to  pilots 

•  issuing  economical  clearances  that  result  in  need  for  few 

additional  instructions  to  handle  aircraft  completely 

•  ensuring  clearances  use  minimum  necessary  flight  path 

changes 

4.  Overall  Safe  and  Efficient  Traffic  Flow  Scale  Rating .  12345678 

n  -  MAINTAINING  ATTENTION  AND  SITUATIONAL  AWARENESS 


5.  Maintaining  Situational  Awareness . 12345678 

•  avoiding  fixation  on  one  area  of  the  radar  scope  when  other 

areas  need  attention 

•  using  scanning  patterns  that  monitor  all  aircraft  on  the  radar 

scope 

6.  Ensuring  Positive  Control . 12345678 

•  tailoring  control  actions  to  situation 

•  using  effective  procedures  for  handling  heavy,  emergency, 
and  unusual  traffic  situations 

7.  Detecting  Pilot  Deviations  from  Control  Instructions .  1  2  3  4  5  6  7  8 

•  ensuring  that  pilots  follow  assigned  clearances  correctly 

•  correcting  pilot  deviations  in  a  timely  manner 

•  ensuring  pilot  adherence  to  issued  clearances 

8.  Correcting  Errors  in  a  Timely  Manner .  12345678 

•  acting  quickly  to  correct  errors 

•  changing  an  issued  clearance  when  necessary  to  expedite 
traffic  flow 

9.  Overall  Attention  and  Situation  Awareness  Scale  Rating .  1  2  3  4  5  6  7  8 
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in -  PRIORITIZING 

10.  Taking  Actions  in  an  Appropriate  Order  of  Importance.. . 1  2  3  4  5  6  7  8 

•  resolving  situations  that  need  immediate  attention  before 

handling  low  priority  tasks 

•  issuing  control  instructions  in  a  prioritized,  structured,  and 

timely  manner 

11.  Preplanning  Control  Actions . . 1  2  3  4  5  6  7  8 

•  scanning  adjacent  sectors  to  plan  for  future  and  conflicting 

traffic 

•  studying  pending  flight  strips  in  bay 

12. _Handling  Control  Tasks  for  Several  Aircraft  12345678 

•  shifting  control  tasks  between  several  aircraft  when  necessary 

•  communicating  in  timely  fashion  while  sharing  time  with  other  actions 

13.  Marking  Flight  Strips  while  Performing  Other  Tasks . 1  2  3  4  5  6  7  8 

•  marking  flight  strips  accurately  while  talking  or  performing 

other  tasks 

•  keeping  flight  strips  current 

14.  Overall  Prioritizing  Scale  Rating . 12345678 

IV  -  PROVIDING  CONTROL  INFORMATION 

15a.  Providing  Essential  Air  Traffic  Control  Information . 12345678 

•  providing  mandatory  services  and  advisories  to  pilots  in  a 

timely  manner 

•  exchanging  essential  information 

ISb.Providing  Additional  Air  Traffic  Control  Information . 1  2  3  4  5  6  7  8 

•  providing  additional  services  when  workload  is  not  a  factor 

•  exchanging  additional  information 

16.  Providing  Coordination . 1  2  3  4  5  6  7  8 

•  providing  effective  coordination 

•  providing  timely  coordination 

•  using  proper  point-out  procedures 

•  performing  hand-off  procedures  properly 

17.  Overall  Providing  Control  Information  Scale  Rating . 12345678 

V  -  TECHNICAL  KNOWLEDGE 

18.  Showing  Knowledge  of  LO As  and  SOPs . 1  2  3  4  5  6  7  8 

•  controlling  traffic  as  depicted  in  current  LOAs 

•  controlling  traffic  as  depicted  in  current  SOPs 

19a.  Showing  Knowledge  of  Aircraft  Capabilities  and  Limitations . 1  2  3  4  5  6  7  8 

•  using  appropriate  speed,  vectoring,  and/or  altitude  assignments 

to  separate  aircraft  with  varied  flight  capabilities 

•  issuing  clearances  that  are  within  aircraft  performance 

parameters 

19b.  Showing  Effective  Use  of  Equipment . 1  2  3  4  5  6  7  8 
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•  updating  of  data  blocks 

•  using  equipment  capabilities 

20.  Overall  Technical  Knowledge  Scale  Rating . 1  2  3  4  5  6  7  8 

VI  -  COMMUNICATING 

21.  Using  Proper  Phraseology . . . 1  2  3  4  5  6  7  8 

•  using  words  and  phrases  specified  in  the  7 1 1 0.65 

•  using  phraseology  that  is  appropriate  for  the  situation 

•  using  minimum  necessary  verbiage 

22.  Communicating  Clearly  and  Efficiently . . 1  234  567  8 


•  speaking  at  the  proper  volume  and  rate  for  pilots  to  understand 

•  speaking  fluently  while  scanning  or  performing  other  tasks 

•  ensuring  clearance  delivery  is  complete 

•  speaking  with  confident,  authoritative  tone  of  voice 


23.  Listening  to  Pilot  Readbacks  and  Requests . 12345678 

•  correcting  pilot  readback  errors 

•  acknowledging  pilot  or  other  controller  requests  promptly 

24.  Overall  Communicating  Scale  Rating . 1  2  3  4  5  6  7  8 
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Appendix  D 
Entry  Questionnaire 


Instructions:  Please  complete  the  form  below.  All  responses  will  be  kept  confidential  and  your  anonymity  is 
guaranteed. 


1.  What  is  your  age?  years 

2.  Are  you  wearing  corrective  lenses  during  this  experiment?  □  Yes  □  No 

3.  How  long  have  you  been  an  FPL  controller?  years 

4.  How  long  have  you  worked  at  your  current  facility?  years 

5.  How  many  months  in  the  past  year  have  you  actively  controlled  traffic?  months 

6.  What  is  your  current  position  as  an  air  traffic  controller?  □  Developmental  □  Full  □  Other: 

Performance 

Level 

7.  Please  list  other  facilities  you  have  worked  at: 

8.  Please  circle  the  number  that  best  describes  your  not  skilled  123456789  10  extremely 

current  skill  as  an  air  traffic  controller.  skilled 

Comments: 

9.  Please  circle  the  number  that  best  describes  the  level  of  no  stress  123456789  10  extremely 

stress  you  have  experienced  during  the  last  several  high  level  of 

months  stress 

Comments: 

10.  Please  circle  the  number  that  best  describes  your  not  motivated  123456789  10  extremely 

motivation  to  participate  in  this  study.  motivated 

Comments: 
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11.  Please  circle  the  number  that  best  describes  your  current  not  healthy  12345678910  extremely 
state  of  health  healthy 


Comments: 


experience  with  video  games.  experienced  experienced 

Comments:  _ _ _ 
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Post-Scenario  Questionnaire  (PSQ) 


1. 

Please  circle  the  number  that  best  describes  how 
realistic  the  simulation  was. 

extremely 

unrealistic 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10  extremely 
realistic 

2. 

Please  circle  the  number  that  best  describes  how 
representative  the  scenario  was  of  a  typical  workday. 

not 

representative 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10  extremely 
representative 

3. 

Please  circle  the  number  that  best  describes  if  the 

ATWIT  device  interfered  with  controlling  traffic. 

no 

interference 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10  extreme 
interference 

4. 

Please  circle  the  number  that  best  describes  if  the 
oculometer  interfered  with  controlling  traffic. 

no 

interference 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10  extreme 
interference 

5. 

Please  circle  the  number  that  best  describes  how  well  the 
simulation-pOots  responded  to  your  clearances  in  terms 
of  traffic  movement  and  call-backs. 

extremely 

poor 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10  extremely 
well 

6. 

Please  circle  the  number  below  that  best  describes  how 
hard  you  were  working  during  this  scenario. 

not  hard 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10  extremely 
hard 

7. 

Please  circle  the  number  that  best  describes  how  well 
you  controlled  traffic  during  this  scenario 

extremely 

poor 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10  extremely 
well 

8. 

Please  circle  the  number  that  best  describes  overall 
situation  awareness  during  this  scenario 

extremely 

poor 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10  extremely 
well 

9. 

Please  circle  the  number  that  best  describes  your 
situation  awareness  for  current  aircraft  locations 
during  this  scenario. 

extremely 

poor 

1 

2 

3 

4 

5 

~6 

7 

8 

T 

10  extremely 
well 

10. 

Please  circle  the  number  that  best  describes  your 
situation  awareness  for  projected  aircraft  locations 
during  this  scenario. 

extremely 

poor 

T 

2 

3 

4 

5 

T 

7 

T 

T 

10  extremely 
well 

11. 

Please  circle  the  number  that  best  describes  your 
situation  awareness  for  potential  violations  during  this 
scenario. 

extremely 

poor 

1 

2 

3 

4 

5 

~6 

7 

8 

9  10  extremely 
well 

12. 

Please  circle  the  number  that  best  describes  how 
difficult  this  scenario  was. 

extremely 

easy 

1 

2 

3 

4 

5 

~6 

7 

8 

9 

10  extremely 
difficult 

Do  you  have  any  other  conunents  about  your  experiences  during  the  simulation? 
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Post-Experimental  Questionnaire 


1.  Please  circle  the  number  that  best  describes  how  realistic  extremely  123456789  10  extremely 

the  simulations  were.  unrealistic  realistic 

Comments: 

2.  Please  circle  the  number  that  best  describes  how  not  123456789  10  extremely 

representative  the  scenarios  were  of  a  typical  workday,  representative  representativ 

e 

Comments: 

3.  Please  circle  the  number  that  best  describes  if  the  no  123456789  10  extreme 

ATWIT  device  interfered  with  controlling  traffic.  interference  interference 

Comments: 

4.  Please  circle  the  number  that  best  describes  if  the  no  123456789  10  extreme 

oculometer  interfered  with  controlling  traffic.  interference  interference 

Comments: 

5.  Please  circle  the  number  that  best  describes  how  well  the  extremely  123456789  10  extremely 

simulation-pilots  responded  to  your  clearances  in  terms  poor  well 

of  traffic  movement  and  call-backs. 

Comments: 

6.  Please  circle  the  number  that  best  describes  if  the  hands-  not  adequate  123456789  10  adequate 
on  training  was  adequate  on  day  1. 

Comments: 

7.  Was  there  anything  that  you  found  particularly  unique  in 
the  simulation  that  you  would  not  see  at  your  home 
facility? 


Comments: 


8.  Were  you  constantly  aware  of  wearing  the  oculometer,  or 
did  you  tune  it  out? 

Comments: 


9.  Do  you  search  the  PVD  in  one  special  way  for 

information?  If  it  depends  on  certain  factors,  what  are 
they? 

Comments: 


10.  Please  circle  the  number  that  best  describes  your 
preference  for  vertical  separation  during  the 
experiment 
Comments: 


no  vertical  123456789  10  always 
separation  vertical 

separation 


11.  Please  circle  the  number  that  best  describes  your 
preference  for  lateral  separation  (i.e.,  ‘S^ectoring^O 
during  the  experiment. 

Comments: 


no  vector  1234567  89  10  always 
separation  vector 

separation 


12.  Please  circle  the  number  that  best  descnbes  your 
preference  for  speed  control  during  the  experiment. 
Comments: 


no  speed  123456789  10  always  speed 
control  control 


13^  Is  there  anything  about  the  study  that  we  should  have 
asked  or  that  you  would  like  to  comment  about? 
Comments: 


Please  circle  the  number  that  best  describes  the 
importance  of  the  following  aircraft  information: 


14.  Aircraft  Call  Sign 

15.  Aircraft  Type 


16.  Aircraft  Beacon  Code 


17.  Controller  Ownership 

18.  Entry  Altitude 


19.  Entry  Airspeed 


extremely  123456789  10  extremely 

_ low _ _ _ high _ 

extremely  1  2345  67  8910  extremely 

low _  high 


extremely  123456789  10  extremely 

_ low  _  high _ 

extremely  123456789  10  extremely 

_ low _ _ _ high _ 

extremely  123456789  10  extremely 

_ low _ _ _ high _ 


extremely  123456789  10  extremely 
low  _ high _ 
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20.  Entry  Fix 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high_. 

21.  Exit  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

22,  Exit  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

23.  Exit  Fix 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

24.  Arrival  Airport  (within  sector) 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

25.  Departure  Airport  (within  sector) 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

26.  Current  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

27.  Current  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

28.  Current  Heading 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

29.  Current  Aircraft  Location 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

30.  Most  Recently  Assigned  Altitude 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

3 1 .  Most  Recently  Assigned  Airspeed 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

32.  Most  Recently  Assigned  Heading 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

33.  Aircraft  Holding/Spinning 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

34.  Aircraft  Waiting  for  Hand-offtRelease 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

T 

10 

extremely 

high 

35.  Aircraft  Near  Exit  Fix/Arrival  Airport 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

T 

10 

extremely 

high 

36.  Density  of  Aircraft  on  Radar  Display 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

T 

T 

10 

extremely 

high 

Please  circle  the  number  that  best  describes  the  importance  of  the 
following  radar  display  information: 

1.  System  Clock 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

2.  VORs 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

3.  Fixes 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

4.  Airports 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

5.  Restricted  Area  Boundaries 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

6.  Runways 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

7.  Sector  Boundaries 

extremely 

low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

extremely 

high 

8.  Filter  Settings 

extremely 

low 

1 
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14.  Vector  Lines 


Appendix  G 
General  Instructions 

Instructions  for  Participants  (given  before  calibration  of  oculometer) 

First  Experimental  Scenario 


Privacy  Statement 

Remember  that  all  data  is  being  collected  without  any  information  which  could  later  be  used  to  identify 
you.  Your  privacy  is  protected. 

Active  Control  histructions  (Scenarios  1  &  4) 

During  this  scenario  we  would  like  for  you  to  control  traffic  as  you  normally  would  in  the  field.  In 
addition,  you  will  be  making  ATWIT  ratings  and  answering  questions  over  the  landline.  A  memory 
recall  procedure  will  occur  after  the  scenario  has  stopped. 

Monitoring  Instructions  (Scenarios  2  &  81 

During  this  scenario  you  will  only  have  to  monitor  the  air  traffic.  Although  there  are  no  pilot/controller 
communications,  you  may  utilize  all  other  normal  control  functions  G-ball,  vector  lines,  route  readouts). 
In  addition,  you  will  be  making  ATWIT  ratings,  answering  questions  over  the  landline.  A  memory 
recall  procedure  will  occur  after  the  scenario  has  stopped. 

ATWrr  Instructions 

One  purpose  of  this  research  is  to  obtain  an  accurate  evaluation  of  controller  workload.  By  workload,  we 
mean  all  the  physical  and  mental  effort  that  you  must  exert  to  do  your  job.  This  includes  maintaining  the 
“picture,”  planning,  coordinating,  decision  making,  communicating,  and  whatever  else  is  required  to 
maintain  a  safe  and  expeditious  traffic  flow.  Every  five  minutes  the  ATWIT  device,  located  to  the  left  of 
the  radar  display,  will  emit  a  brief  tone  and  ten  buttons  will  appear.  The  buttons  will  remain  visible  for 
only  a  limited  amount  of  time.  The  way  you  will  tell  us  how  hard  you  are  working  is  by  pushing  the 
buttons  numbered  from  1  to  10  which  will  appear  on  the  ATWIT. 

I  will  review  what  these  buttons  mean  in  terms  of  your  workload.  At  the  low  end  of  the  scale  (1  or  2), 
your  workload  is  low  -  you  can  accomplish  everything  easily.  As  the  numbers  increase,  your  workload  is 
getting  higher.  Numbers  3, 4,  and  5  represent  increasing  levels  of  moderate  workload  where  the  chance 
of  error  is  still  low  but  steadily  increasing.  Numbers  6, 7,  and  8  reflect  relatively  high  workload  where 
there  is  some  chance  of  making  errors.  At  the  high  end  of  the  scale  are  numbers  9  and  10,  which 
represent  a  very  high  workload,  where  it  is  likely  that  you  will  have  to  leave  some  tasks  unfinished. 

All  controllers,  no  matter  how  proficient  and  experienced,  will  be  exposed  at  one  time  or  another  to  all 
levels  of  workload.  It  does  not  detract  from  a  controller’s  professionalism  when  he  indicates  that  he  is 
working  very  hard  or  that  he  is  hardly  working.  Feel  free  to  use  the  entire  scale  and  tell  us  honestly  how 
hard  you  are  working.  Do  not  sacrifice  the  safe  and  expeditious  flow  of  traffic  in  order  to  respond  to  the 
ATWn  device.  Remember,  your  workload  rating  should  not  reflect  how  much  you  are  working  during 
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the  course  of  the  scenario.  Instead,  your  rating  should  reflect  how  much  workload  you  are  experiencing 
during  the  instant  when  you  are  prompted  to  make  the  rating. 

Do  you  have  any  questions  about  using  the  ATWIT  device? 

SPAM  Instmctions 

A  single  landline  will  be  used  for  all  coordination  purposes  during  the  experiment.  In  addition  to 
coordination  activities,  at  various  times  during  the  scenario  you  will  receive  a  call  over  the  landline  from 
‘Tech  Center.”  During  the  call  you  will  be  asked  a  question  and  will  be  given  two  response  options. 
Please  answer  each  question  as  quickly  and  accurately  as  possible.  In  answering  each  question  you  may 
use  any  and  all  information  normally  available  to  you  including  the  radar  scope  and  flight  progress 
strips. 

Do  you  have  any  questions  about  using  answering  questions  over  the  landline? 

Recall  Procedure 

After  the  scenario  has  been  stopped  you  will  perform  a  memory  recall  procedure.  You  will  see  a 
representation  of  the  airspace  on  your  display.  Within  the  airspace  will  be  the  raw  radar  returns,  vector 
lines,  and  leader  lines  as  they  appeared  when  the  scenario  ended.  At  the  bottom  of  the  display  will  be  a 
bin  containing  the  data  blocks  from  all  of  the  aircraft  that  were  in  your  airspace  or  otherwise  under  your 
control  when  the  scenario  was  stopped.  Your  task  is  to  move  the  data  blocks  from  the  bin  to  their 
respective  position  in  the  airspace  as  quickly  and  accurately  as  possible. 

To  place  a  data  block,  select  the  data  block  from  the  bin  by  using  the  left  button  on  the  trackball.  The 
data  block  will  change  color  when  it  is  selected.  After  selecting  a  data  block  from  the  bin,  move  the 
cursor  to  the  appropritate  position  and  push  the  left  button  to  place  the  data  block.  Once  a  datablock  has 
been  placed  it  will  change  color  in  the  bin  (from  green  to  gray).  To  remove  a  data  block  that  has  already 
been  placed,  select  the  placed  data  block  using  the  left  trackball  button.  Once  selected,  the  data  block 
will  be  highlighted  in  the  bin.  Move  the  cursor  over  the  highlited  datablock  in  the  bin  and  press  the  left 
trackball  button.  The  data  block  will  move  back  into  the  bin. 

Remember  to  complete  the  data  block  placements  as  quickly  and  accurately  as  possible. 

Do  you  have  any  questions  before  we  calibrate  the  oculometer? 
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Instructions  for  Participants  (given  before  calibration  of  oculometer) 

Subsequent  Scenarios 

Privacy  Statement 

Remember  that  all  data  is  being  collected  without  any  information  which  could  later  be  used  to  identify 
you.  Your  privacy  is  protected. 

Active  Control  Instructions  (Scenarios  1  &  4) 

During  this  scenario  we  would  like  for  you  to  control  traffic  as  you  normally  would  in  the  field.  In 
addition,  you  will  be  making  ATWIT  ratings  and  answering  questions  over  the  landline.  A  memory 
recall  procedure  will  occur  after  the  scenario  has  stopped. 

Monitoring  Instructions  (Scenarios  2  &  8) 

During  this  scenario  you  will  only  have  to  monitor  the  air  traffic.  Although  there  are  no  pilot/controller 
communications,  you  may  utilize  all  other  normal  control  functions  (j-ball,  vector  lines,  route  readouts). 
In  addition,  you  will  be  making  ATWIT  ratings,  answering  questions  over  the  landline.  A  memory 
recall  procedure  will  occur  after  the  scenario  has  stopped. 

ATWIT  Instructions 

One  purpose  of  this  research  is  to  obtain  an  accurate  evaluation  of  controller  workload.  By  workload,  we 
mean  whatever  physical  and  mental  effort  you  must  exert  to  maintain  safe  and  expeditious  traffic  flow. 
Buttons  numbered  from  1  to  10  will  appear  on  the  screen  to  your  left.  Push  the  button  which  describes 
your  current  level  of  workload.  At  one  extreme,  numbers  1  and  2,  represent  low  workload  -  you  can 
accomplish  everything  easily.  At  the  other,  numbers  9  and  10  represent  a  very  high  workload,  where  it 
is  likely  that  you  will  have  to  leave  some  tasks  unfinished. 

All  controllers,  no  matter  how  proficient  and  experienced,  will  be  exposed  at  one  time  or  another  to  all 
levels  of  workload.  It  does  not  detract  from  a  controller’s  professionalism  when  he  indicates  that  he  is 
working  very  hard  or  that  he  is  hardly  working.  Feel  free  to  use  the  entire  scale  and  tell  us  honestly  how 
hard  you  are  working.  Do  not  sacrifice  the  safe  and  expeditious  flow  of  traffic  to  respond  to  the  ATWIT 
device.  Remember,  your  workload  rating  should  not  reflect  how  much  you  are  working  during  the 
course  of  the  scenario.  Instead,  your  rating  should  reflect  how  much  workload  you  are  experiencing 
during  the  instant  when  you  are  prompted  to  make  the  rating. 

Do  you  have  any  questions  about  using  the  ATWIT  device? 

SPAM  Instructions 

You  will  receive  calls  over  the  landline  from  ‘Tech  Center.”  During  the  call  you  will  be  asked  a 
question  and  will  be  given  two  response  options.  Please  answer  each  question  as  quickly  and  accurately 
as  possible.  In  answering  each  question  you  will  be  allowed  to  use  any  and  all  information  normally 
available  to  you  including  the  radar  scope  and  flight  progress  strips. 

Do  you  have  any  questions  about  using  answering  questions  over  the  landline? 

Do  you  have  any  questions  before  we  calibrate  the  oculometer? 
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Recall  Instractions  (given  immediately  after  end  of  scenario) 

In  a  moment  you  will  see  a  representation  of  the  airspace.  Within  the  airspace  will  be  the  raw  radar 
returns,  vector  lines,  and  leader  lines  as  they  appeared  when  the  scenario  ended.  Place  the  data  blocks 
from  the  bin  in  their  respective  positions  in  the  airspace  as  quickly  and  accurately  as  possible. 

Do  you  have  any  questions  about  the  recall  procedure? 
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Appendix  H 
Visual  Scanning 


H.l  Visual  Scanning  Variables 


The  oculometer  recorded  eye  movements  during  both  practice  scenarios  and  experimental 
scenarios.  H-1  provides  a  sunmiary  of  the  eye  movement  measures. 

Table  H-1.  Visual  Scanning  Variables 


1. 

'Conditional  information  -  Aircraft 

21. 

2. 

Conditional  information  -  Location 

22. 

3. 

Conditional  information  -  Range 

23. 

4. 

Conditional  information  -  Tightness 

24. 

5. 

Eye  motion  workload 

25. 

6. 

Pupil  motion  workload 

26. 

7. 

Visual  efficiency 

27. 

8. 

Mean  number  of  fixations 

28. 

9. 

Mean  duration  of  fixations 

29. 

10. 

Mean  fixation  area 

30. 

11. 

Mean  distance  of  saccades 

31. 

12. 

Mean  duration  of  saccades 

32. 

13. 

Mean  number  of  dwells 

33. 

14. 

Mean  dwell  area 

34. 

15. 

Mean  duration  of  dwells 

35. 

16. 

Number  of  fixations  on  target 

36. 

17. 

Mean  duration  of  fixations  on  target 

37. 

18. 

Number  of  fixations  off  target 

38. 

19. 

20. 

Mean  duration  of  fixations  off  target 

Number  of  Fixations  on  radar  returns 

39. 

Mean  duration  of  fixations  on  radar  returns 

Number  of  fixations  on  data  blocks 

Mean  duration  of  fixations  on  data  blocks 

Number  of  fixations  on  other  static  objects 

Mean  duration  of  fixations  on  other  static  objects 

Number  of  fixations  on  PVD 

Mean  duration  of  fixations  on  PVD 

Number  of  fixations  on  SCRD 

Mean  duration  of  fixations  on  SCRD 

Number  of  fixations  on  map 

Mean  duration  of  fixations  on  map 

Number  of  fixations  on  flight  strips 

Mean  duration  of  fixations  on  flight  strips 

Number  of  fixations  on  keyboard 

Mean  duration  of  fixations  on  keyboard 

Number  of  fixations  on  trackball 

Mean  dxiration  of  fixations  on  trackball 

Number  of  fixations  on  ATWIT 

Mean  duration  of  fixations  on  ATWIT 
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H.1.1  Fixations 


A  fixation  is  a  sequence  of  at  least  6  oculometer  samples  with  an  intersample  distance  of  less 
than  1  degree  of  visual  angle.  At  1  meter  distance  this  corresponds  to  a  circle  with  a  8.73  mm 
radius.  The  distance  between  two  samples  is  the  norm  of  the  vectorial  difference  of  the  sample 
coordinates.  If  2  fixations  are  not  separated  by  either  a  blink  or  a  saccade  (see  definitions 
below),  these  fixations  should  be  combined  within  one  fixation.  In  summary: 

Fixation  if: 

D  =  V((xi-Xi+i)^  +(yi-yi+i)^)  >  8.73  mm 

with  D  the  distance  between  to  subsequent  samples  x  and 
y  the  horizontal  and  vertical  point  of  gaze  coordinates  in 
mm  respectively 

and: 

n  >  6  with  n  the  number  of  samples  in  a  sequence 

and 

separated  by  a  blink  or  a  saccade 

Related  to  a  fixation  the  following  variables  need  to  be  calculated:  Fixation  Duration  and 
Fixation  Area.  Fixation  Area  is  an  approximation  of  the  area  covered  by  the  POG  due  to  eye 
movements  within  a  fixation. 

Fixation  Duration: 

FDCDUR  =  tsampie  *  Ssamples 

with  tsampie  where  the  duration  of  a  sample  (Veo  second) 
and  Ssample  is  the  total  number  of  samples  within  a 
fixation 

Fixation  Area: 

FIXAREA  =  (max(xfix)-min(xfix))*(max(yf,x)-min(yf,x)) 

with  Xflx  and  ynx  the  sequences  of  horizontal  and  vertical 
POG  coordinates  within  a  fixation  respectively 

H.1.2  Blink 

A  blink  is  the  complete  or  partial  closure  of  the  eye.  The  oculometer  will  suggest  that  the 
velocity  at  the  start  and  end  of  a  blink  was  greater  than  700  degrees  per  second  which 
corresponds  with  6. 108  "*4.  This  is  physically  impossible,  but  it  does  give  us  a  way  to  determine 
start  and  end  of  a  blink.  A  blink  starts  after  the  last  sample  of  the  previous  fixation  and  stops 
before  the  first  sample  of  the  next  fixation.  In  summary: 

Blink  if: 

VEL  =  ■\/((Xi-Xj+i)  +(yi"yi+l)  )/ tsampie  >6.108  /s 

with  VEL  being  the  a  crude  estimate  of  the  tangential 
velocity  and  x  and  y  the  horizontal  and  vertical  point  of 
gaze  coordinates  in  mm  respectively.  The  index  denotes 
the  current  sample  i  and  next  sample  i+1  respectively 

and: 

n  >12  with  n  the  number  of  samples  in  a  sequence 
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Related  to  a  blink  the  following  variables  need  to  be  calculated:  Fixation  Duration  and  Blink 
Distance.  Blink  Distance  is  the  distance  covered  by  the  POG  due  to  eye  movements  during  a 
blink. 

Blink  Duration: 

BLNKDUR  =  tsampie  *  Zsamples 

with  tsampie  where  the  duration  of  a  sample  ( Veo  second) 
and  Ssample  is  the  total  number  of  samples  within  a  blink 

Blink  Distance: 

BLNKDST  =  (x„-xp)*(y„-yp) 

with  X  and  y  the  horizontal  and  vertical  point  of  gaze 
coordinates  in  mm  respectively.  The  index  denotes  the 
last  sample  of  the  previous  fixation  p  and  first  sample  of 
the  next  fixation  n  respectively 

H.1.3  Saccade 

A  saccade  is  the  ballistic  movement  of  the  eye  from  one  fixation  to  the  next.  A  saccade  is 
characterized  by  fast  eye  movements  of  up  to  700  degrees  per  second.  The  cut-off  for  a  saccade 
is  a  difference  in  distance  between  two  subsequent  saccades  that  is  greater  or  equal  to  8.73  mm, 
lasts  at  least  3  samples  (or  a  velocity  of  0.524  “/s),  and  the  velocity  is  less  or  equal  to  700  degrees 
per  second  (6.108  “/s).  The  saccade  will  start  at  the  end  of  the  last  sample  of  the  previous 
fixation  and  will  end  at  the  beginning  of  the  first  sample  of  the  next  fixation.  In  surmnary: 

0.524  >VEL>  6.108“/* 

and: 

n>2 

Related  to  saccades  a  number  of  variables  need  to  be  calculated:  Saccade  Duration,  Saccade 
Distance,  and  Saccade  Velocity.  The  saccade  distance  is  the  angular  distance  traveled  during  a 
saccade  in  degrees.  The  saccade  velocity  is  the  average  velocity  within  a  saccade  in  degrees  per 
second. 

Saccade  Duration: 

SACDUR=  tsampie  *  Ssamples 

with  tsampie  where  the  duration  of  a  sample  (Veo  second) 
and  Zsample  is  the  total  number  of  samples  within  a 
saccade 

Saccade  Distance: 

SACDST  =  (x„-Xp)*(y„-yp) 

with  X  and  y  the  horizontal  and  vertical  point  of  gaze 
coordinates  in  mm  respectively.  The  index  denotes  the 
last  sample  of  the  previous  fixation  p  and  first  sample  of 
the  next  fixation  n  respectively 

Saccade  Velocity: 
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SACVEL  —  2  (V((Xi-Xi+i)  +(yi"yi+l)  ))  /  tsample  *  Usaccade 

with  tsample  where  the  duration  of  a  sample  (Veo  second) 
and  nsaccade  is  the  number  of  samples  within  the  saccade 

H.1.4  Dwell 

A  dwell  is  defined  as  a  sequence  of  fixations  that  return  to  a  location  within  1  degree  of  visual 
from  a  target  location  or  within  1  degree  of  visual  angle  if  the  POG  does  not  rest  on  a  target. 
This  way  included  in  a  dwell  are  also  moving  targets. 

Related  to  dwells  a  number  of  variables  need  to  be  calculated:  Dwell  Duration  and  Dwell  Area. 
Dwell  Duration  is  the  duration  between  the  start  of  the  first  sample  of  the  first  fixation  and  the 
end  of  the  last  sample  of  the  last  fixation  within  a  dwell  sequence.  Dwell  Area  is  an 
approximation  of  the  area  covered  by  the  POG  within  a  dwell. 

Dwell  Duration: 

DDUR  =  tn,fix  m  "  1 

with  ti,fix  1  is  the  start  of  the  first  sample  of  the  first 
fixation  and  tn,fix  m  is  the  end  (sample  n)  of  the  last 
fixation  (fixation  m). 

Dwell  Area: 

DAREA  =  (max(Xf5x)-min(Xflx))*(max(yfix)-min(yf,x)) 

with  Xfix  and  ygx  the  sequences  of  horizontal  and  vertical 
POG  coordinates  within  a  dwell  respectively 


H.1.5  Visual  Efficiency 

Visual  efficiency  is  defined  as  the  proportion  of  the  total  scanning  time  that  is  spent  fixating. 
Visual  Efficiency: 

VISEFF  =  (mean(FIXDUR)  *  Nfix)  / 

(mean(FIXDUR)  *  Nfu  +  mean(S  ACDUR)  *  Nsac) 

In  fact,  this  is  nothing  more  than  the  portion  of  the  time  that  the  eye  is  fixed  once  the  blinks  are 
removed: 

Visual  Efficiency: 

VISEFF  =  2:FIXDUR/(2:FrXDUR  +  i:S  ACDUR) 

with  ZFIXDUR  the  sum  of  the  duration  of  the  fixations, 
ZSACDUR  the  sum  of  the  duration  of  the  saccades  and 
TIME  the  total  time  in  seconds. 
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H.1.6  Eve  Motion  Workload 

Eye  Motion  Workload  is  defined  as  the  average  saccade  motion  in  degrees  by  the  number  of 
saccades,  or; 

Eye  Motion  Workload: 

EYEMWL  =  mean  (S  ACDST)  *  N^ac  /  TIME 

with  Nsac  the  number  of  saccades  within  the  interval 
under  study  and  TIME  the  total  time  in  seconds. 

In  fact,  this  is  nothing  more  than  the  total  distance  traveled  divided  by  the  total  the  time: 

Eye  Motion  Workload: 

EYEMWL  =  ZSACDST/TIME 

with  ZS  ACDST  the  sum  of  the  distance  of  the  saccades 
in  degrees  and  TIME  the  total  time  in  seconds. 

H.1.7  Pupil  Motion  Workload 

Pupil  Motion  Workload  is  defined  as  the  sum  of  the  average  pupil  diameter  within  a  fixation 
divided  by  the  total  time  within  the  interval  under  consideration. 

Pupil  Motion  Workload 

PUPMWL  =  Ellmean(PUPDIAM)fixi-mean(PUPDIAM)fixi+i)ll/TIME 

with  PUPDIAM  the  pupil  diameter  in  mm  based  on  a 
conversion  from  ASL  arbitrary  units  to  mm  of  0.044  mm 
per  ASL  unit.  The  index  fix  i  and  fix  i+1  denote  the  i-th 
and  the  i+lth  fixation  respectively 

It  seems  if  the  author  of  the  article  that  this  measure  was  based  on  was  after  the  “distance” 
traveled  during  an  interval.  I  is  oif  course  possible  to  separate  the  oculometer  samples  that  do  not 
include  blinks  and  then  to  calculate  the  cumulative  sum  of  the  pupil  diameter  differences.  This 
may  be  a  more  accurate  estimate  of  pupil  workload: 

Pupil  Average  Work: 

for  fixations  or  saccades: 

PUPAW  =  ZIIPUPDIAMi  -  PUPDIAMi+ill 

with  i  and  i+1  oculometer  sample  i  an  i+1  respectively, 
hi  this  case  the  oculometer  samples  that  occur  during 
blinks  are  removed  from  the  timeseries  of  data. 


H.1.8  Conditional  Information 


The  conditional  information  is  defined  by  Brillouin  (1962)  as  described  in  Ellis  (1986).  The 
formula  will  here  be  given  without  getting  too  much  into  the  details: 
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CONINF  =  E  Pi  *  [E  pij  *  log2  (pij)]  with  i 

with  Pi  is  simple  probability  of  viewing  target  i,  and  py  is 
the  probability  of  a  transition  from  target  i  to  target  j. 
Simple  probability  was  defined  by  Ellis  (1986)  as  the 
percentage  of  time  spent  on  each  particular  target  or 
jumping  between  each  target.  Here  we  will  calculate  it 
not  as  a  percentage  of  time,  but  the  ratio  of  the  number  of 
times  on  a  target  and  the  total  number  of  fixations  and  the 
number  of  transitions  and  the  total  number  of  saccades 
for  Pi  and  pi j  respectively. 


H.2.  Visual  Scanning:  Inferential  Statistics 

Table  H-2.  MANOVA  Results  for  Saccade 
Duration  and  Distance,  Fixation  Number,  Duration,  and  Area 


Wilks’ 

Lambda 

RaoR 
Form  2 

mgm 

V 

df  1 

df2 

p-level 

Task  Load 

.326 

4.558 

0.674 

4.558 

5 

11 

.017 

Involvement 

.612 

1.394 

0.388 

1.394 

5 

11 

.299 

Task  Load  x  Involvement 

.874 

0.316 

0.126 

0.316 

5 

11 

.893 

Table  H-3.  ANOVA  Results  for  Eye  Movement  Related  Variables  (p  <  .01) 


Means  sqr 
Effect 

Means  sqr 
Error 

F(l,15) 

/?-level 

Saccade  duration 

Load 

0.001 

0.000 

.027 

Involvement 

0.001 

0.000 

KLlsisi 

.028 

Load  X  Involvement 

0.000 

0.000 

0.152 

.702 

Saccade  Distance 

Load 

0.066 

0.185 

.559 

Involvement 

0.053 

0.177 

.593 

Load  X  Involvement 

0.177 

0.127 

1.400 

.255 

Fixation  Number 

Load 

163248.578 

81323.086 

2.007 

Involvement 

55676.969 

82758.891 

0.673 

HEBI 

Load  X  Involvement 

25214.367 

107323.008 

0.235 

.635| 

Fixation  Duration 

Load 

0.004 

3.454 

Involvement 

0.001 

0.944 

Load  X  Involvement 

0.000 

0.002 

0.196 

.664 

Fixation  Area 

Load 

0.000 

0.005 

0.044 

.837 

Involvement 

0.001 

0.008 

0.075 

.787 

Load  X  Involvement 

0.000 

0.005 

0.021 

.887 
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Table  H-4.  MANOVA  Results  for  Blink  Number,  Blink  Duration,  and  Pupil  Diameter 


Wilks' 

Lambda 

Rao  R 
Form 

Pillai- 

Bartlett 

Trace 

1 

dfl 

df2 

/?-level 

Task  Load 

.760 

1.371 

0.240 

1.371 

3 

13 

.295 

Involvement 

.500 

4.328 

0.500 

4.328 

3 

13 

Task  Load  x  Involvement 

.873 

0.631 

0.127 

0.631 

3 

13 

■ESQ 

Table  H-5.  ANOVA  Results  for  Visual  Scanning  Related  Workload  Indicators  (p  <  .017) 


Effect 

Error 

F(l,15) 

/7-level 

Blink  Number 

Load 

12064.542 

10835.198 

1.113 

0.308 

Involvement 

7295.088 

13280.576 

0.549 

0.470 

Load  X  Involvement 

7553.572 

8055.369 

0.938 

0.348 

Blink  Duration 

Load 

0.011 

Involvement 

0.002 

Load  X  Involvement 

0.001 

0.002 

0.870 

0.366| 

Pupil  Diameter 

Load 

0.079 

0.160 

Involvement 

0.347 

0.090 

hb 

Load  X  Involvement 

0.026 

0.296 

0.088 

0.771| 

Table  H-6.  MANOVA  Results  for  Fixation  Number,  Duration,  and  Area  by 

Load,  Involvement  and  Time 


Wilks’ 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

V 

dfl 

df2 

/7-level 

Load 

■IB 

1.868 

0.301 

1.868 

3 

13 

Involvement 

■QE 

1.841 

1.841 

3 

13 

■EE] 

Time 

353.723 

353.723 

15 

1 

Load  X  Involvement 

0.056 

0.013 

3 

13 

Load  X  Time 

11.696 

0.994 

15 

1 

Involvement  x  Time 

.014 

4.535 

0.986 

4.535 

15 

1 

.355 

Load  X  Involvement  x  Time 

.055 

1.146 

0.945 

1.146 

15 

1 

.635 
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Table  H-7.  ANOVA  Results  for  Interval-Based  Eye  Movement  Related  Variables  (p  <  .017) 


Means  sqr 
Effect 

Means  sqr 
Error 

F(dfl,2) 

dfl 

df2 

p-level 

Number 

Load 

5107.655 

3995.906 

1 

m 

.276 

Involvement 

138.047 

5709.520 

1 

■s 

.879 

Time 

11589.951 

838.325 

13.825 

4 

60 

.000 

Load  X  Involvement 

400.655 

4525.971 

1 

15 

Load  X  Time 

2576.260 

889.213 

4 

60 

Involvement  x  Time 

1718.002 

1101.729 

4 

Wail 

.182 

Load  X  Involvement  x  Time 

1292.058 

612.508 

4 

.073 

Duration 

Load 

0.031 

0.005 

6.187 

1 

15 

.025 

Involvement 

0.009 

1 

15 

.221 

Time 

beesi 

0.001 

4 

60 

.000 

Load  X  Involvement 

0.000 

0.008 

0.013 

1 

15 

.912 

Load  X  Time 

0.002 

0.002 

1.164 

1 

m 

Involvement  x  Time 

0.004 

0.002 

2.349 

■! 

MSB! 

Load  X  Involvement  x  Time 

0.001 

0.001 

1.134 

■ 

60 

.350] 

Area 

Load 

0.003 

0.034 

1 

15 

Involvement 

0.004 

0.049 

1 

15 

Time 

0.067 

0.009 

bee 

4 

60 

.000 

Load  X  Involvement 

0.001 

0.029 

1 

15 

.886 

Load  X  Time 

0.014 

0.009 

1.523 

4 

60 

.193 

Involvement  x  Time 

0.001 

4 

■ 

Load  X  Involvement  x  Time 

0.009 

■B 

4 

m 

n 
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Table  H-8.  Saccade  Characteristics:  MANOVA  Results 


MANOVA,  adjusted 
alpha=0.0169 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

df  1 

df2 

/7-level 

Load 

.475 

7.733 

0.525 

7.733 

2 

14 

.005 

Interval  1 

.936 

0.478 

0.478 

2 

14 

■IB 

Interval  2 

.760 

2.211 

II^QQ 

2 

14 

mm 

Interval  3 

.385 

11.177 

2 

14 

.001 

Interval  4 

.622 

4.256 

14 

.036 

Interval  5 

.449 

8.595 

■E 

14 

.004 

Interval  6 

.774 

2.044 

0.226 

2.044 

2 

14 

.166 

Involvement 

.662 

3.568 

0.338 

3.568 

14 

Interval  1 

.967 

0.236 

0.033 

0.236 

m 

Interval  2 

.649 

3.791 

0.351 

3.791 

2 

■Q 

.048 

Interval  3 

mm 

0.586 

9.906 

2 

14 

.002 

Interval  4 

mm 

0.379 

4.271 

2 

14 

.036 

Interval  5 

.861 

1.128 

0.139 

1.128 

2 

14 

.351 

Interval  6 

mm 

0.296 

2.949 

2 

.085 

Time 

wns^ 

0.922 

7.076 

10 

6 

.013 

Low  Load,  Monitoring 

.177 

2.785 

0.823 

2.785 

10 

6 

.111 

Low  Load,  Active 

.019 

30.276 

10 

6 

.000 

High  Load,  Monitoring 

.237 

1.932 

mESE^ 

10 

6 

.217 

High  Load,  Active 

.309 

1.340 

0.691 

1.340 

10 

6 

.374 

Load  X  Involvement 

.944 

0.413 

0.056 

0.413 

2 

.669 

Load  X  Time 

.259 

1.718 

0.741 

1.718 

10 

6 

.262 

Involvement  x  Time 

.170 

2.920 

0.830 

2.920 

10 

6 

.101 

Load  X  Involvement  x  Time 

.099 

5.459 

0.901 

5.459 

10 

6 

.025 
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Table  H-9.  Saccade  Duration:  ANOVA  Results,  Interval  Beised 


Duration 

Means  sqr 
Effect 

Means  sqr 
Error 

f(dfl,2) 

dfl 

df2 

/7-level 

Load 

0.007 

0.001 

9.227 

1 

15 

.008 

Monitoring 

0.005 

0.001 

4.449 

1 

15 

.052 

Active  Control 

0.002 

0.001 

1.775 

1 

15 

.203 

Interval  1 

0.000 

0.000 

0.924 

1 

15 

.352 

Interval  2 

0.001 

0.000 

4.679 

1 

15 

.047 

Interval  3 

0.004 

0.000 

14.777 

1 

15 

.002 

Interval  4 

0.000 

0.000 

0.877 

1 

15 

.364 

Interval  5 

0.005 

0.001 

9.875 

1 

15 

.007 

Interval  6 

0.000 

0.000 

1.134 

1 

15 

.304 

Involvement 

0.009 

0.001 

6.778 

1 

15 

.020 

Low  Load 

0.007 

0.002 

3.942 

1 

15 

.066 

High  Load 

0.003 

0.001 

2.382 

1 

15 

.144 

Interval  1 

0.000 

o.ooo" 

0.010 

1 

15 

.923 

Interval  2 

0.004 

0.001 

7.494 

1 

15 

.015 

Interval  3 

0.003 

0.000 

11.484 

1 

15 

.004 

Interval  4 

0.002 

0.000 

9.141 

1 

15 

.009 

Interval  5 

0.001 

0.000 

1.206 

1 

15 

.289 

Interval  6 

0.002 

0.000 

6.170 

1 

15 

.025 

Time 

0.001 

0.000 

4.911 

5 

75 

.001 

Low  Load,  Monitoring 

0.000 

0.000 

2.401 

5 

75 

.045 

Low  Load,  Active 

0.001 

0.000 

4.494 

5 

75 

.001 

High  Load,  Monitoring 

0.002 

0.000 

7.479 

5 

75 

.000 

High  Load,  Active 

0.000 

0.000 

1.871 

5 

75 

.109 

Load  X  Involvement 

0.000 

0.002 

0.255 

1 

15 

.621 

Load  X  Time 

0.001 

0.000 

4.091 

5 

75 

.002 

Involvement  x  Time 

0.000 

0.000 

2.578 

5 

75 

.033 

Load  X  Involvement  x  Time 

0.001 

0.000 

7.656 

5 

75 

.000 
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Table  H- 10.  Saccade  Distance:  ANOVA  Results,  Merval  Based 


Distance 

Means 
sqr  Effect 

Means  sqr 
Error 

F(dfl,2) 

dfl 

df2 

p-level 

Load 

0.473 

1.139 

0.415 

m 

B 

.529 

Monitoring 

1.005 

0.974 

1.032 

B 

B 

.326 

Active  Control 

0.001 

0.822 

0.001 

B 

B 

BB] 

Involvement 

0.932 

1.097 

0.849 

B 

B 

bb 

Low  Load 

1.437 

2.988 

B 

B 

HESI 

High  Load 

0.028 

0.022 

B 

B 

Time 

0.601 

4.257 

B 

B 

Load  X  Involvement 

0.533 

0.812 

B 

B 

Load  X  Time 

0.331 

0.129 

E 

B 

Involvement  x  Time 

0.147 

0.131 

IHBIQ 

B 

B 

Load  X  Involvement  x  Time 

0.291 

0.154 

1.887 

B 

B 

.107| 

Table  H-12.  Fixation  Characteristics  by  Scene  Plane  :  MANOVA  Results 


MANOVA,  adjusted  alpha=0.0253 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

dfl 

df2 

/7-level 

Task  Load 

.669 

3.465 

0.331 

3.465 

2 

14 

.060 

Scene  Plane 

3090.435 

1.000 

309.435 

m 

2 

.003 

Task  Load  x  Involvement 

■Em 

0.133 

0.019 

0.133 

m 

14 

.876 

Task  Load  x  Scene  Plane 

EESm 

0.946 

2.498 

14 

2 

.322 

Involvement  x  Scene  Plane 

0.942 

2.326 

14 

2 

.341 

Task  Load  x  Involvement  x  Scene  Plane 

.163 

0.731 

0.837 

0.731 

14 

2 

.713 

Table  H-13.  Number  of  Fixations  by  Scene  Plane :  ANOVA  Results 


Number 

Means  sqr 
Effect 

Means  sqr 
Error 

F(dfl,2) 

dfl 

df2 

/7-level 

Task  Load 

20406.072 

10165.386 

2.007 

B 

15 

.177 

Flight  Strip  Bay 

34929.801 

35612.473 

0.981 

B 

■B 

.338 

Keyboard 

64983.879 

5993.706 

10.842 

B 

m 

.005 

Track  Ball 

2.680 

251.643 

0.011 

1 

15 

.919 

ATwrr 

44.804 

302.532 

0.148 

1 

15 

.706 

CRD/QAK 

24132.621 

2069.708 

1 

15 

.004 

Map 

192.180 

95.695 

eeeei 

B 

15 

All 

Land  Line 

554.147 

243.046 

2.280 

1 

15 

HEm 

Task  Load  x  Scene  Plane 

7615.680 

12110.361 

0.629 

6 

90 

HEm 

Involvement  x  Scene  Plane 

16867.813 

13762.795 

Eu^S 

B 

90 

.295 

Task  Load  x  Involvement  x  Scene  Plane 

10782.088 

11363.794 

eeb 

B 

90 

.473 
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Table  H-14.  Fixation  Duration  by  Scene  Plane:  ANOVA  Results 


Duration 

Means  sqr 

Means  sqr 

Effect 

Error 

Task  Load 

0.057 

0.019 

Flight  Strip  Bay 

0,001 

0.001 

Keyboard 

bhb 

0.006 

Track  Ball 

0.162 

ATwrr 

0.000 

0.006 

CRD/QAK 

0.164 

0.005 

Map 

0.187 

0.010 

Land  Line 

0.035 

0.017 

Task  Load  x  Scene  Plane 

0.004 

0.027 

Involvement  x  Scene  Plane 

0.111 

0.027" 

Task  Load  x  Involvement  x  Scene  Plane 

0.002 

0.028 

F(dfl,2)  dfl  df2  /7-level 


Table  H-15.  Fixation  Characteristics  by  Scene  Plane:  MANOVA  Results,  Interval  Based 


MANOVA,  adjusted  alpha=0.0253 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

dfl 

dfl 

p-level 

Load 

.519 

6.498 

6.498 

m 

14 

.010 

Scene 

.042 

157.927 

157.927 

WB 

14 

.000 

Load  X  Involvement  _ 


Load  X  Time 


Involvement  x  Time  _ 


Load  X  Scene 


Involvement  x  Scene 


Time  x  Scene  _ 


Load  X  Time  x  Scene 


Involvement  x  Time  x  Scene 


Load  X  Involvement  x  Time  x  Scene 


Table  H-16.  Number  of  Fixations  by  Scene  Plane:  ANOVA  Results,  Interval  Based 


Number 

Means  sqr 
Effect 

Means  sqr 
Error 

F(dfl,2) 

Load 

3.255 

2515.722 

0.001 

Scene 

7051866.500 

52165.211 

135.183 

Load  X  Involvement 

1665.574 

1972.219 

0.845 

Load  X  Time 

1289.157 

563.348 

2.288 

Involvement  x  Time 

586.443 

600.980 

0.976 

Load  X  Scene 

2051.382 

5271.664 

0.389 

Involvement  x  Scene 

807.317 

8940.859 

0.090 

Time  x  Scene 

22828.459 

1751.238 

13.036 

Load  X  Time  x  Scene 

160.772 

1248.355 

0.129 

Involvement  x  Time  x  Scene 

5021.814 

1505.616 

3.335 

Load  X  Involvement  x  Time  x  Scene 

8122.203 

1250.881 

6.493 

Table  H-17.  Fixation  Duration  by  Scene  Plane;  ANOVA  Results,  Interval  Based 


Duration 

Means 

sqr 

Effect 

Means 

sqr 

Error 

F(dfl,2) 

1,15 

dfl 

df2 

/7“level 

Load 

0.073 

0.006 

11.291 

1 

15 

.004 

Scene 

7.604 

0.026 

293.290 

6 

90 

.000 

Load  X  Involvement 

0.007 

0.009 

0.788 

1 

15 

.389 

Load  X  Time 

0.004 

0.003 

1.106 

5 

75 

Involvement  x  Time 

0.003 

0.002 

1.740 

5 

75 

Load  X  Scene 

0.004 

0.003 

1.106 

6 

.310 

Involvement  x  Scene 

0.004 

0.003 

1.376 

6 

.259 

Time  x  Scene 

0.016 

0.003 

5.727 

30 

.000 

Load  X  Time  x  Scene 

0.007 

0.003 

30 

.025 

Involvement  x  Time  x  Scene 

0.002 

0.003 

30 

90 

Load  X  Involvement  x  Time  x  Scene 

0.002 

0.003 

30 

90 

Table  H- 18.  Fixation  Characteristics  by  Radarscope  Object: 
MANOVA  Results,  Scenario  Based 


MANOVA,  adjusted  alpha=0.0253 

Rao  R 
Form  2 

Pillai- 

Bardett 

Trace 

■ 

dfl 

df2 

Task  Load 

.398 

10.587 

2 

14 

.002 

Task  Load  x  Involvement 

.862 

1.116 

2 

14 

.335 

Task  Load  x  Object 

.383 

2.686 

6 

10 

.081 

Involvement  x  Object 

.491 

1.728 

2 

10 

.212 

Load  X  Involvement  x  Object 

.697 

0.724 

2 

10 

.641 

H-13 


Table  H-19.  Fixation  Characteristics  by  Radarscope  Object:  MANOVA  Results,  Interval  Based 


MANOVA,  adjusted  alpha=0.0253 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

dfl 

df2 

/?-level 

Load 

.755 

2.268 

2 

14 

BB!] 

Involvement 

.824 

bee 

bsee 

1.496 

2 

14 

EB^ 

Radar  Return 

0.211 

bee 

2 

14 

IBEB 

Data  Block 

■naw 

0.274 

b^e 

14 

^BS 

Time 

Biaiiki 

bjlhi 

6 

bezq 

Object 

■rjaia 

Bi&iy 

BOSS 

2 

14 

^BSS 

Monitoring 

.446 

M;fAlrJ 

0.554 

BISS 

2 

14 

BESS 

Active  Control 

.634 

0.366 

bsb 

14 

ebsq 

Load  X  Involvement 

BESS 

BSE 

bh 

bee 

BB 

14 

ebb 

Load  X  Time 

bee 

BQ 

6 

^ESi 

Involvement  x  Object 

.762 

be 

0.238 

2.189 

BB 

14 

^BSI 

Time  x  Object 

.278 

■resa 

0.722 

Bi^ 

BE 

6 

Load  X  Involvement  x  Time 

.036 

0.964 

BE 

^BSS 

Load  X  Involvement  x  Object 

BBSS] 

bed 

0.091 

2 

bq 

.513 

Load  X  Time  x  Object 

HE 

bee 

0.387 

B!^ 

10 

BB 

.916 

Involvement  x  Time  x  Object 

1.004 

1.004 

10 

6 

B^S 

Load  X  Involvement  x  Time  x  Object 

1.762 

BiUS 

1.762 

10 

6 

EI^B 

Table  H-20.  Number  of  Fixations  by  Radarscope  Object:  ANOVA  Results 


Number 

Means  sqr 
Effect 

Means  sqr 
Error 

F(dfl,2) 

dfl 

df2 

/7-level 

Load 

718.561 

1388.644 

0.517 

1 

15 

^bb 

Involvement 

885.017 

3852.203 

0.230 

1 

15 

BfllBl 

Time 

2285.364 

515.099 

BSai 

BE 

75 

bh 

Object 

49889.352 

6274.989 

EIBSI 

■D 

15 

.013 

Load  X  Involvement 

41.054 

1699.183 

beh 

15 

.879 

Load  X  Time 

498.905 

322.769 

ebbs 

5 

75 

.186 

Involvement  x  Time 

1744.968 

541.476 

be^e 

5 

75 

.011 

Load  X  Object 

418.541 

914.811 

bezb 

1 

15 

.509 

Involvement  x  Object 

14356.219 

3427.043 

4.189 

1 

15 

.059 

Time  x  Object 

841.015 

819.159 

1.027 

5 

75 

.408 

Load  X  Involvement  x  Time 

3377.578 

369.996 

bsee 

5 

75 

^BiSS] 

Load  X  Involvement  x  Object 

49.247 

1265.070 

beeb 

15 

Load  X  Time  x  Object 

314.950 

0.692 

BE 

75 

.631 

Involvement  x  Time  x  Object 

646.686 

555.013 

1.165 

■B 

75 

.334 

Load  X  Involvement  x  Time  x  Object 

719.144 

306.276 

2.348 

_ 5 

75 

.049 

H-14 


Table  H-21.  Fixation  Duration  by  Radarscope  Object:  ANOVA  Results 


Duration 

Means 

sqr 

Effect 

Means 

sqr 

Error 

F(dfl,2) 

dfl 

df2 

p-level 

Load 

0.050 

0.011 

4.391 

1 

wm 

HE 

Involvement 

0.052 

0.030 

1.747 

1 

■s 

■1^ 

Time 

0.040 

0.003 

5 

75 

Object 

0.013 

0.006 

1 

15 

■BO] 

Load  X  Involvement 

0.002 

0.021 

1 

15 

.789 

Load  X  Time 

0.013 

0.005 

5 

75 

.022 

Involvement  x  Time 

0.005 

0.004 

1.328 

5 

75 

.262 

Load  X  Object 

0.001 

0.004 

0.224 

1 

15 

.643 

Involvement  x  Object 

0.003 

0.003 

1.265 

1 

15 

.278 

Time  x  Object 

0.002 

0.002 

0.854 

5 

75 

.516 

Load  X  Involvement  x  Time 

0.002 

0.003 

0.661 

5 

75 

.654 

Load  X  Involvement  x  Object 

0.003 

0.002 

1 

■& 

.296 

Load  X  Time  x  Object 

0.000 

0.002 

HESQ 

5 

.989 

Involvement  x  Time  x  Object 

0.002 

0.002 

1.265 

5 

75 

Load  X  Involvement  x  Time  x  Object 

0.001 

0.002 

0.384 

5 

75 

Table  H-22.  Conditional  Information  bidices:  MANOVA  Results 


MANOVA,  adjusted 
alpha=0.0126 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

dfl 

df2 

1 

Ci, 

Load 

.133 

19.602 

0.867 

19.602 

4 

12 

.000 

Monitoring 

.306 

6.816 

0.694 

6.816 

4 

12 

.004 

Active  Control 

■BiES 

17.934 

BQE 

17.934 

m 

m 

.000 

Involvement 

■Bl^ 

13.158 

fiia'gl 

13.158 

4 

m 

.000 

Low  Load 

1.527 

BBB3 

1.527 

m 

12 

.256 

High  Load 

.104 

25.734 

bei^ 

25.734 

4 

12 

.000 

Load  X  Involvement 

.156 

16.172 

0.844 

16.172 

4 

12 

.000 

Table  H-23.  Object-Based  Conditional  Information  Index:  ANOVA  Results 


Duration 

Means 

sqr 

Effect 

Means 
sqr  Error 

F(dfl,2) 

dfl 

df2 

p-level 

Load 

0.008 

0.000 

54.332 

i 

■s 

.000 

Monitoring 

0.002 

0.000 

9.947 

1 

wm 

.007 

Active  Control 

0.007 

0.000 

76.643 

1 

15 

.000 

Involvement 

0.001 

0.000 

6.336 

1 

15 

.024 

Low  Load 

0.000 

0.000 

0.063 

1 

15 

.806 

High  Load 

0.002 

0.000 

24.556 

1 

15 

.000 

Load  X  Involvement 

0.001 

0.000 

8.413 

1 

15 

.011 

H-15 


H.3.  Visual  Scanning:  Scenario  Based  Descriptive  Statistics 

Table  H-24.  Saccade  I>uration:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Saccade  Duration 
(msec) 

Low 

E3Sa 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

111 

22 

120 

16 

116 

19 

Active  Control 

121 

19 

127 

19 

124 

19 

116 

21 

124 

17 

120 

19 

Table  H-25.  Saccade  Distance:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Saccade 

Distance 

(inches) 

Low 

Hi 

■ 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

3.595 

0.525 

3.772 

0.498 

Active  Control 

3.757 

0.346 

3.716 

WkVSSi 

0.514 

3.676 

0.445 

3.742 

0.564 

3.708 

0.503 

Table  H-26.  Eye  Motion  Workload:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Eye  Motion 
Workload 
(-)  . 

Low 

■v 

■ 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

1.199 

ilirJLii 

1.033 

Active  Control 

bob 

0.715 

0.816 

5.510 

0.993 

5.809 

0.838 

5.655 

0.926 

Table  H-27.  Number  of  Fixations:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Number  of 
Fixations 

Low 

mi 

Mean 

SD 

Mean 

SD 

Mean 

SD  1 

Monitoring 

2699 

2846 

281 

Active  Control 

2798 

299 

2859 

238 

2749 

410 

2853 

255 

2799 

345| 

Table  H-28.  Fixation  Duration:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Fixation 

Duration 

(msec) 

Low 

■Mi 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

Hsn 

71 

430 

52 

436 

62 

Active  Control 

44 

416 

43 

426 

44 

438 

58 

423 

47 

431 

53 

H-16 


Table  H-29.  Fixation  Area:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Fixation  Area 

Low 

Hi! 

(sq.  inches) 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

0.663 

0.186 

0.663 

0.186 

0.663 

0.183 

Active  Control 

0.660 

0.119 

0.653 

0.115 

0.657 

0.115 

0.662 

0.154 

0.658 

0.149 

0.660 

0.150 

Table  H-30.  Visual  Scanning  Efficiency:  Mean  and  Standard  Deviations  by  Load  and 

Involvement 


Visual  Scanning 
Efficiency 
(-) 

Low 

Hi 

■ 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

0.795 

0.051 

0.781 

0.039 

0.789 

0.046 

Active  Control 

0.782 

0.037 

0.765 

0.033 

itWZI 

0.035 

0.789 

0.045 

0.773 

0.036 

0.781 

0.041 

Table  H-3 1.  Number  of  Dwells:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Number  of 
Dwells 

Low 

High 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

2481 

473 

2611 

258 

2541 

387 

Active  Control 

2531 

276 

2615 

265 

2573 

269 

2506 

382 

2613 

257 

2558 

329 

Table  H-32.  Dwell  Duration:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Dwell  Duration 
(msec) 

Low 

High 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

482 

79 

472 

66 

477 

72 

Active  Control 

489 

53 

463 

58 

476 

56 

486 

66 

467 

61 

477 

64 

Table  H-33.  Dwell  Area:  Mean  and  Standard  Deviations  by  Load  and  Involvement 


Dwell  Area 
(sq.  inches) 

Low 

High 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Monitoring 

0.777 

0.203 

0.772 

0.188 

0.775 

0.193 

Active  Control 

0.784 

0.128 

0.759 

0.128 

0.771 

0.127 

0.780 

0.167 

0.765 

0.156 

0.773 

0.161 

H-17 


Table  H-34.  Number  of  Blinks:  Means  and  Standard  Deviations  by  Load  and  Involvement 


Number  of 
Blinks 

Low 

High 

Means 

SD 

Means  SD  | 

Monitoring 

218 

390 

194 

Active  Control 

161 

413 

158 

388 

164 

417 

188 

402 

175 

Table  H-35.  Blink  Duration:  Means  and  Standard  Deviations  by  Load  and  Involvement 


Blink  Duration 
(msec) 

Low 

High 

Means 

SD 

Means  SD  | 

Monitoring 

256 

58 

238 

55 

248 

56 

Active  Control 

218 

43 

236 

78 

255  80 

227  49 

241  68  1 

Table  H-36.  Pupil  Diameter:  Means  and  Standard  Deviations  by  Load  and  Involvement 


Pupil  Diameter 
(mm) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD  1 

Monitoring 

6.548 

0.985 

Active  Control 

6.628 

0.931 

6.658 

0.847 

6.534 

0.877 

6.569 

0.881 1 

H-18 


Table  H-37.  Number  of  Fixations  by  Scene  Plane  by  Load  and  Involvement 


Number  of  Fixations 
by  Scene  Planes 

Scene 

Plane 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

RSN 

1808 

423 

1898 

308 

1850 

370 

FSN 

713 

362 

769 

275 

739 

320 

KBN 

48 

75 

52 

52 

50 

64 

TBN 

8 

12 

14 

9 

13 

ATN 

23 

9 

23 

6 

23 

8 

CRN 

78 

39 

67 

46 

73 

42 

MDN 

10 

13 

3 

6 

7 

11 

LLN 

13 

13 

24 

37 

18 

27 

Active  Control 

RSN 

1839 

247 

1868 

170 

FSN 

721 

232 

660 

226 

KBN 

87 

100 

145 

133 

116 

119 

TBN 

3 

4 

15 

26 

9 

19 

ATN 

29 

20 

7 

25 

23 

CRN 

96 

■■El 

130 

83 

113 

64 

MDN 

10 

10 

10 

14 

10 

12 

LLN 

12 

12 

11 

11 

12 

11 

RSN 

1823 

341 

1882 

241 

1852 

296 

FSN 

717 

299 

711 

251 

714 

KBN 

67 

101 

112 

84 

101 

TBN 

6 

9 

13 

21 

9 

16 

ATN 

26 

■■E 

21 

6 

24 

17 

CRN 

87 

llllllllllll^^ 

101 

74 

94 

58 

MDN 

10 

11 

7 

11 

9 

11 

LLN 

13 

12 

17 

27 

15 

21 

H-19 


Table  H-38.  Fixation  Duration  (msec)  by  Scene  Plane  by  Load  and  Involvement 


Number  of  Fixations  by 
Scene  Planes 


1  Low 

High 

iMeans 

SD 

Means 

SD 

5021  72| 


308  51 


244  164 


305  743 


331  86| 


392  105 


174  183 


492 


301 


226  92 


Scene  Plane 


Radar  Scope 


Flight  Strip  Bay 


Keyboard 


Track  Ball 


ATwrr 


CRD/QAK 


Map 


Land  Line 


Radar  Scope 


Flight  Strip  Bay 


Keyboard 


Track  Ball 


ATwrr 


CRD/QAK 


Map 


Land  Line 


Radar  Scope 


Flight  Strip  Bay 


Keyboard 


Track  Ball 


ATwrr 


CRD/QAK 


Map 


Land  Line 


Table  H-39.  Number  of  Fixations  by  Radarscope  Object  by  Load  and  Involvement 


222  132 


223 


328 


445 


215  169 


M\ 


210  73 


176 


321 


410 


167 


120 


374 

99 

141 

164 

267 

154 

486 

56 

297 

30 

198 

72 

124 

126 

322 

85 

479 

70 

252 

155 

218 

103 

492 

•MSB! 

301 

216 

107 

msmm 

324 

428 

Witil 

198 

167 

242 

131 

Scene  Planes 


Monitoring 


Radar  Scope  Object 

1  Low  1 

1  High  1 

_ 

Mean 

SD 

Mean 

SD 

System  Area 

4 

7 

3 

5 

3 

6 

Other  Static  Objects 

19 

11 

12 

14 

16 

13 

Radar  Returns 

787 

206 

827 

118 

806 

169 

Data  Blocks 

926 

258 

987 

250 

955 

252 

System  Area 

2 

2 

1 

1 

1 

1 

Other  Static  Objects 

19 

12 

19 

18 

19 

15 

Radar  Returns 

867 

125 

866 

98 

866 

110 

Data  Blocks 

900 

146 

924 

99 

912 

123 

System  Area 

3 

5 

2 

3 

2 

4 

Other  Static  Objects 

19 

11 

15 

16 

17 

14 

Radar  Returns 

827 

172 

848 

107 

837 

144 

Data  Blocks 

913 

207 

954 

184 

933 

196 

H-20 


Table  H-40.  Number  of  Fixations  by  Radar  Scope  Object  by  Load  and  Involvement 


Number  of  Fixations  by 
Radar  Scope  Objects 

Radar  Scope  Objects 

Low 

High 

Means 

133 

Means 

133 

Means 

133 

Monitoring 

System  Area 

4 

m 

3 

5 

3 

6 

Other  Static  Objects 

19 

11 

12 

14 

16 

13 

Radar  Returns 

787 

206 

827 

118 

806 

IKS 

Data  Blocks 

926 

258 

987 

250 

955 

Active 

Control 

System  Axea 

2 

2 

1 

1 

■B 

1 

Other  Static  Objects 

19 

12 

19 

18 

15 

Radar  Returns 

867 

125 

866 

98 

866 

no 

Data  Blocks 

900 

146 

924 

99 

912 

123 

System  Area 

3 

5 

2 

3 

2 

4 

Other  Static  Objects 

19 

m 

15 

16 

■EB 

14 

Radar  Returns 

827 

172 

848 

107 

WKSSl 

144 

Data  Blocks 

913 

207 

954 

184 

933 

196 

Table  H-41.  Fixation  Duration  by  Radarscope  Object  by  Load  and  Involvement 


Fixation  Duration  by 
Radar  Scope  Objects 

Radar  Scope  Objects 

Low 

High 

Means 

SD 

Means 

SD 

Monitoring 

System  Area 

336 

277 

185 

228 

265 

263| 

Other  Static  Objects 

236 

54 

MSB 

Radar  Returns 

513 

74 

Data  Blocks 

514 

86 

497 

mm 

506 

73 

Active 

Control 

System  Area 

291 

270 

60 

108 

175 

234 

Other  Static  Objects 

213 

65 

193 

71 

Radar  Returns 

61 

493 

57 

502 

58 

Data  Blocks 

495 

58 

483 

63 

489 

60 

System  Area 

313 

270 

183 

Other  Static  Objects 

224 

60 

kb 

100 

mm 

Radar  Returns 

66 

499 

62 

506 

64| 

Data  Blocks 

73 

489 

60 

Table  H-42.  Object-Based  Conditional  Information  Index  Object  by  Load  and  Involvement 


COB 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

0.151 

0.017 

0.135 

0.007 

0.144 

0.015 

Active  Control 

0.152 

0.010 

0.121 

0.009 

0.137 

0.018 

0.151 

0.014 

0.128 

0.011 

0.140 

o 

b 

H-21 


Table  H-43.  Range-Based  Conditional  Information  Index  by  Load  and  Livolvement 


CRA 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD  1 

Monitoring 

1.002 

Active  Control 

0.998 

QQII 

0.992 

0.014 

1.000 

0.012 

H.3.2  Interval  -  Based 

Table  H-44.  Number  of  Fixations  by  Load,  Involvement,  and  Time 


Number  of 
Fixations 


W) 

a 


a 

o 


c 

o 

U 

■I 


1  Low 

1  High 

Means 

lag 

Means 

laa 

Means 

Bil 

502 

58 

506 

469 

HI 

m 

Eli 

468 

46 

465 

m 

^ESI 

B 

m 

H 

m 

458 

48 

m 

B33 

S 

iiliiii 

481 

El 

is 

487 

53 

465 

MSB 

486 

48 

475 

57 

469 

m 

482 

49 

E[S 

467 

469 

51 

IBl^ 

m 

461 

46 

462 

464 

45 

1 

illfi 

1  491 

BSSl 

500 

m 

496 

55 

HI 

E! 

493 

480 

57 

m 

472 

51 

470 

48 

1  456 

m 

470 

50 

463 

50 

\wm 

m 

m 

B 

E 

1  465 

48 

El 

1 

L_Ci 

H-22 


Table  H-45.  Fixation  Duration  by  Load,  Livolvement,  and  Time 


Fixation 

Duration 

(msec) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

397 

66 

388 

47 

393 

57 

450 

83 

425 

46 

438 

68 

464 

70 

440 

62 

452 

66 

464 

78 

450 

61 

457 

69 

450 

56 

445 

77 

447 

66 

475 

87 

442 

57 

459 

74 

'  '  ■■■  ■'5 

11 

Ml 

Active  Control 

412 

54 

396 

43 

404 

49 

451 

62 

404 

51 

428 

61 

429 

48 

401 

51 

416 

50 

447 

53 

430 

57 

439 

55 

440 

50 

433 

57 

437 

52 

444 

61 

iPi 

64 

i 

kMiSI 

EQ3 

mm 

ESI 

451 

72 

414 

49 

433 

64 

446 

61 

421 

59 

434 

61 

455 

66 

440 

59 

448 

63 

445 

52 

439 

67 

442 

59 

459 

75 

440 

60 

450 

68 

iSi 

ilMii 

ISI 

H-23 


Table  H-46.  Fixation  Area  by  Load,  Involvement,  and  Time 


Fixation  Area 

Low 

Hi 

gh 

(inch2) 

Means 

SD 

Means 

SD 

Means 

SD 

0.607 

0.160 

0.619 

0.146 

0.613 

0.151 

0.677 

0.211 

0.589 

0.169 

0.635 

0.194 

bO 

0.716 

0.226 

0.657 

0.212 

0.687 

0.217 

1 

0.685 

0.286 

0.674 

0.165 

0.679 

0.230 

C 

o 

0.680 

0.209 

0.735 

0.194 

0.708 

0.200 

0.666 

0.199 

ItfcVi;! 

1 

m 

MMi 

1 

rs 

0.619 

0.142 

0.618 

0.129 

0.626 

0.152 

0.626 

0.120 

0.626 

0.135 

0.689 

0.143 

0.660 

0.120 

0.675 

0.131 

u 

0.659 

0.132 

0.661 

0.124 

0.660 

0.126 

0.691 

0.112 

0.698 

0.174 

0.694 

0.143 

0.688 

0.132 

0.688 

0.121 

mm 

Mm 

ii 

0.613 

0.149 

0.618 

0.131 

0.615 

0.139 

0.651 

0.183 

0.608 

0.145 

0.630 

0.166 

0.702 

0.185 

0.659 

0.170 

0.681 

0.177 

0.672 

0.217 

0.667 

0.144 

0.669 

0.183 

0.686 

0.163 

0.717 

0.182 

0.701 

0.172 

0.677 

0.159 

0.696 

0.214 

0.686 

0.187 

gllFi 

11 

H-24 


Table  H-47.  Visual  Efficiency  by  Load,  Involvement,  and  Time 


BBSii 

Low 

msm 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

0.766 

0.063 

0.759 

0.053 

0.763 

0.058 

0.806 

0.053 

0.801 

0.032 

0.803 

0.043 

0.815 

0.040 

0.781 

0.047 

0.798 

0.046 

0.802 

0.043 

0.806 

0.043 

0.804 

0.042 

0.810 

0.042 

0.757 

0.056 

0.783 

0.056 

liSi 

0.784 

0.049 

0.798 

0.046 

liiil 

lili 

MM\ 

Active  Control 

0.773 

0.043 

0.762 

0.042 

0.768 

0.043 

0.796 

0.047 

0.758 

0.041 

0.778 

0.048 

0.780 

0.045 

0.750 

0.036 

0.765 

0.043 

0.789 

0.037 

0.769 

0.035 

0.779 

0.037 

0.780 

0.038 

0.765 

0.038 

0.772 

0.038 

0.772 

0.776 

0.040 

0.042 

ill 

iliiiil 

0.770 

0.054 

0.760 

0.047 

0.765 

0.050 

0.801 

0.049 

0.779 

0.042 

0.791 

0.047 

0.797 

0.046 

0.766 

0.044 

0.782 

0.047 

0.795 

0.040 

0.788 

0.043 

0.792 

0.041 

0.794 

0.042 

0.761 

0.047 

0.778 

0.048 

0.791 

0.047 

0.780 

0.044 

0.785 

0.046 

11^ 

11 

mm 
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Table  H-48.  Number  of  Dwells  by  Load,  Involvement,  and  Time 


H-26 


Table  H-49.  Dwell  Duration  by  Load,  bivolvement,  and  Time 


Dwell 

Duration 

(msec) 

Low 

High 

Means 

SD 

|Means 

SD  1 

Means 

SD 

Monitoring 

432 

76 

mm 

430 

68 

BBM 

Hsai 

480 

78 

513 

77 

486 

89 

499 

83 

510 

87 

491 

74 

501 

80 

490 

65 

482 

486 

78 

521 

106 

486 

65  1 

503 

88 

_ 

vuk  J 

Active  Control 

456 

67 

441 

60 

449 

63 

506 

74 

447 

67 

477 

75 

67 

444 

70 

464 

w 

58 

479 

73 

491 

55 

491 

67 

509 

77 

1  493 

81  1 

501 

78 

1 

M: 

y 

444 

71 

mm 

439 

66 

84 

gCT 

\wm 

mm 

76 

gSM 

72 

grdJi 

81 

78 

72 

gf!f 

72 

496 

73 

59 

mm 

489 

72 

QQ 

91 

msM 

502 

82 

iieii 

H-27 


Table  H-50.  Dwell  Area  by  Load,  Involvement,  and  Time 


Dwell 

Area 

(sq.  inches) 

Low 

Hi 

Means 

SD 

Means 

SD 

Means 

SD 

0.708 

0.163 

0.724 

0.139 

0.716 

0.149 

0.786 

0.222 

0.708 

0.173 

0.748 

0.201 

eJ  . 

‘C 

0.847 

0.251 

0.764 

0.228 

0.806 

0.240 

Q 

0.812 

0.314 

0.776 

0.164 

0.794 

0.247 

a 

o 

0.792 

0.236 

0.837 

0.215 

0.815 

0.223 

i< 

0.782 

0.216 

0.816 

0.293 

0.799 

0.253 

StSB 

0.721 

0.147 

0.732 

0.126 

0.726 

0.135 

■g 

0.738 

0.165 

0.721 

0.123 

0.730 

0.144 

o 

0.815 

0.157 

0.748 

0.119 

0.783 

0.142 

U 

0.790 

0.158 

0.766 

0.159 

0.779 

0.156 

cu 

•a 

0.827 

0.136 

0.824 

0.214 

0.826 

0.175 

0.831 

0.111 

0.808 

0.151 

0.820 

0.130 

imii 

0.714 

0.153 

0.728 

0.130 

0.721 

0.141 

0.762 

0.194 

0.715 

0.148 

0.739 

0.174 

0.831 

0.205 

0.756 

0.179 

0.794 

0.195 

0.801 

0.242 

0.771 

0.159 

0.786 

0.204 

0.810 

0.189 

0.831 

0.211 

0.820 

0.198 

0.807 

0.169 

0.812 

0.229 

0.810 

0.199 

gill 

iii^ 
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Table  H-51.  Saccade  Duration  by  Load,  Livolvement,  and  Time 


SDUM 

(msec) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

119 

29 

123 

26 

121 

27 

107 

27 

105 

15 

106 

21 

103 

20 

124 

23 

113 

24 

112 

23 

109 

16 

111 

20 

106 

23 

141 

28 

123 

31 

108 

19 

121 

23 

114 

21 

IMS 

1 

o 

U 

1 

119 

19 

124 

24 

122 

21 

115 

23 

130 

20 

122 

23 

121 

25 

133 

20 

127 

23 

117 

20 

128 

20 

123 

20 

128 

21 

131 

22 

129 

22 

124 

19 

127 

21 

IIS 

119 

24 

124 

25 

121 

24 

111 

25 

118 

21 

114 

23 

113 

24 

128 

22 

120 

24 

115 

21 

118 

20 

117 

21 

117 

24 

136 

25 

126 

26 

119 

23 

122 

21 

121 

22 

^11 
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Table  H-52.  Saccade  Distance  by  Load,  Involvement,  and  Time 


SDIM 
(inch  2) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

3.686 

0.737 

3.935 

0.829 

3.806 

0.780 

3.437 

0.726 

3.533 

0.489 

3.483 

0.614 

3.648 

0.636 

3.861 

0.619 

3.754 

0.626 

3.683 

0.657 

3.472 

0.475 

3.577 

0.574 

3.491 

0.392 

3.974 

0.746 

3.733 

0.635 

3.561 

0.504 

3.638 

0.545 

3.599 

0.517 

iHi 

IsSM] 

fitilii 

Active  Control 

3.739 

0.456 

3.750 

0.747 

3.744 

0.604 

3.554 

0.471 

3.794 

3.670 

0.646 

3.711 

0.561 

3.902 

0.681 

3.803 

0.619 

3.770 

0.465 

wm 

0.734 

3.712 

0.602 

4.003 

0.450 

3.873 

0.673 

3.940 

0.563 

3.797 

0.469 

3.596 

0.715 

3.699 

0.600 

US 

1* 

Hi 

3.712 

0.603 

3.842 

0.781 

3.775 

0.692 

3.495 

0.605 

3.663 

0.659 

3.577 

0.632 

3.680 

0.589 

3.882 

0.640 

3.779 

0.618 

3.728 

0.558 

3.561 

0.614 

3.646 

0.588 

3.755 

0.491 

3.924 

0.700 

3.838 

0.604 

3.682 

0.493 

3.617 

0.625 

3.650 

0.558 
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Table  H-53.  Eye  Motion  Workload  by  Load,  Involvement,  and  Time 


Eye  Motion 
Workload 
(inch/sec) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

5.724 

1.343 

6.106 

1.183 

5.909 

1.262 

5.312 

1.201 

5.839 

0.789 

5.567 

1.041 

5.703 

1.097 

5.819 

0.850 

5.761 

0.966 

5.567 

1.059 

5.331 

0.871 

5.449 

0.961 

5.478 

0.646 

5.819 

0.944 

5.648 

0.814 

5.442 

0.969 

5.626 

0.939 

5.534 

0.942 

3'j|j 

Active  Control 

5.545 

1.164 

5.811 

1.335 

5.674 

1.236| 

5.433 

0.829 

6.007 

1.244 

5.711 

5.769 

1.046 

6.221 

1.123 

5.988 

1.090 

5.930 

0.836 

5.621 

0.852 

5.780 

0.844 

5.973 

0.784 

5.978 

0.950 

5.975 

0.854 

5.730 

0.878 

5.554 

1.008 

5.645 

0.931 

1 

L'kiiSJ 

Vi-i ;  i 

5.635 

1.240 

5.959 

1.248 

5.791 

1.244 

5.372 

1.017 

5.923 

1.027 

5.639 

1.051 

5.737 

1.054 

6.020 

1.000 

5.876 

1.029 

5.754 

0.952 

5.476 

0.859 

5.617 

0.911 

5.733 

0.752 

5.899 

0.934 

5.814 

0.843 

5.590 

0.919 

5.590 

0.958 

5.590 

.2: 

a 

illra’l 
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Table  H-54.  Blink  Number  by  Load,  Involvement,  and  Time 


Table  H-55.  Blink  Duration  by  Load,  Involvement  and  Time 


Blink 

Duration 

(msec) 


Means 

SD 

Means 

SD 

278 

96 

257 

86 

251 

78 

240 

63 

236 

43 

234 

56 

247 

51 

221 

44 

253 

61 

237 

46 

255 

65 

229 

46 

mmmm 

263 

106 

214 

43 

254 

113 

207 

43 

258 

112 

203 

42 

262 

111 

211 

48 

236 

81 

209 

42 

laa 

Hi 

ill 

96 

250 

79 

257 

92 

227 

54 

245 

86 

221 

51 

253 

86 

212 

43 

258 

89 

224 

48 

246 

73 

219 

44 

258 

88 

242 

77 

233 

71 

233 

71 

241 

73 

233 

62 

Table  H-56.  Pupil  Diameter  by  Load,  Involvement,  and  Time 


Pupil 

Diameter 

(mm) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

6.5 

0.8 

6.5 

0.9 

6.5 

0.8 

6.5 

0.9 

6.6 

1.0 

6.5 

0.9 

6.4 

'  0.9 

6.6 

1.0 

6.5 

0.9 

6.3 

0.9 

6.6 

1.0 

6.5 

0.9 

.„9 

6.6 

0.9 

6.5 

0.9 

hb 

0.9 

6.6 

1.0 

6.5 

0.9 

IS 

ilM 

i®i 

1^ 

Active  Control 

6.6 

0.9 

6.5 

0.9 

6.6 

0.9 

6.6 

0.9 

6.6 

0.9 

6.6 

6.6 

0.9 

6.6 

0.9 

MB! 

wm 

0.9 

0.9 

0.9 

6.6 

0.9 

1^^ 

0.8 

moQ 

0.9 

hi  ' 

6.6 

0.9 

6.5 

0.9 

6.6 

0.9 

6.6 

0.9 

6.6 

0.9 

6.6 

0.9 

6.5 

0.9 

6.6 

0.9 

6.6 

0.9 

6.5 

0.9 

6.7 

0.9 

6.6 

0.9 

6.5 

0.9 

6.6 

0.9 

6.5 

0.9 

6.5 

0.9 

0.9 

6.6 

0.9 

SUB 

iMlI 
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Table  H-57.  Number  of  Fixations  on  a  Target  by  Load,  Involvement,  and  Time 


Number  of 
Fixations  on  a 
Target 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

495 

62 

499 

54 

497 

57 

463 

60 

495 

51 

478 

58 

461 

45 

455 

54 

458 

49 

434 

49 

467 

49 

450 

51 

464 

m 

439 

48 

452 

48 

450 

3 

460 

53 

455 

53 

etMa 

R 

Active  Control 

479 

61 

491 

44 

485 

53 

462 

65 

482 

49 

472 

58 

465 

47 

478 

50 

471 

48 

461 

53 

464 

51 

463 

51 

456 

53 

452 

47 

454 

49 

447 

53 

452 

51 

449 

51 

495 

49 

491 

55 

462 

62 

488 

50 

475 

57 

463 

45 

467 

53 

465 

49 

448 

52 

466 

49 

457 

51 

460 

49 

446 

47 

453 

48 

449 

53 

456 

51 

452 

52 

in 

mm 

wm 
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Table  H-58.  Percentage  of  Fixations  on  Target  by  Load,  Involvement,  and  Time 


Percentage  of 
Fixations  on 
Target 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

99 

2 

99 

1 

99 

2 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

97 

3 

99 

1 

98 

2 

98 

2 

99 

1 

99 

1 

98 

2 

98 

2 

98 

2 

ISSsil  mSsIi 

Active  Control 

100 

0 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

99 

0 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

98 

1 

98 

3 

98 

2 

99 

1 

98 

6 

98 

4 

liEii 

lilii 

99 

2 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

99 

1 

98 

2 

99 

1 

99 

2 

98 

1 

99 

2 

98 

2 

99 

2 

98 

4 

98 

3 

iSjjj 
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Table  H-59.  Duration  of  Fixations  on  Target  by  Load,  Involvement,  and  Time 


Fixation 
Duration  on 
Target 
(msec) 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

404 

74 

394 

48 

399 

62 

457 

87 

430 

46 

444 

70 

471 

70 

447 

65 

459 

68 

477 

78 

454 

61 

465 

70 

459 

58 

449 

78 

454 

68 

484 

88 

450 

60 

467 

iii 

fMM 

MM 

Hsii 

Active  Control 

414 

54 

399 

wm 

407 

49 

455 

64 

407 

52 

431 

62 

433 

48 

405 

53 

419 

51 

453 

53 

435 

58 

444 

55 

447 

50 

442 

60 

445 

54 

449 

58 

451 

72 

450 

64 

1 

Mm 

409 

64 

396 

45 

403 

55 

456 

75 

418 

50 

438 

66 

451 

62 

426 

62 

439 

63 

464 

66 

444 

59 

455 

63 

453 

54 

446 

68 

449 

61 

466 

75 

450 

65 

458 

70 

SiSI 

ia 
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Table  H-60.  Number  of  Fixations  not  on  Target  by  Load,  Involvement,  and  Time 
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Table  H-61.  Fixation  Area  by  Load,  Involvement,  and  Time 


Fixation  Area 

Low  1 

(inch2) 

Means 

SD 

Means 

SD 

Means 

SD 

0.607 

0.160 

0.619 

0.146 

0.613 

0.151 

0.677 

0.211 

0.589 

0.169 

0.635 

0.194 

bO 

C 

‘C 

0.716 

0.226 

0.657 

0.212 

0.687 

0.217 

o 

0.685 

0.286 

0.674 

0.165 

0.679 

0.230 

o 

0.680 

0.209 

0.735 

0.194 

0.708 

0.200 

0.666 

0.199 

0.704 

0.279 

0.685 

0.239 

Sii 

1*1 

0.619 

0.142 

0.617 

0.119 

0.618 

0.129 

0.626 

0.152 

0.626 

0.120 

0.626 

0.135 

o 

0.689 

0.143 

0.660 

0.120 

0.675 

0.131 

U 

o 

0.659 

0.132 

0.661 

0.124 

0.660 

0.126 

> 

•a 

0.691 

0.112 

0.698 

0.174 

0.694 

0.143 

< 

0.688 

0.115 

0.688 

0.132 

0.688 

0.121 

ilSSli 

0.613 

0.149 

0.618 

0.131 

0.615 

0.139 

0.651 

0.183 

0.608 

0.145 

0.630 

0.166 

0.702 

0.185 

0.659 

0.170 

0.681 

0.177 

0.672 

0.217 

0.667 

0.144 

0.669 

0.686 

0.163 

0.717 

0.182 

0.701 

0.172 

0.677 

0.159 

0.696 

0.214 

0.686 

0.187 

; 

(iMM 

Hiiil 

Table  H-62.  Number  of  Fixations  by  Scene  Planes  by  Load,  Involvement,  and  Time 
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Land  Line  I  Map  Diaplay  |  Soft  CRD  I  ATWIT  I  Track  Ball  I  Keyboard  I  Flight  Strip  Bay  I  Radarscope 


Table  H-63.  Fixation  Duration  by  Scene  Planes  by  Load,  Involvement,  and  Time 


Monitoring 


Active  Control 


Collapsed  across  Involvement 


IgSBlMSB  lijnSHIMKlB  ETOIMBI  rSlCTliE?] 

IK^SilES  lE^^llES  lEjSSillll^ 

EQK!  K^B  K^H  Ban  ESIH]  BSn 

BiSnilB^Ma^^MSlK3^EK5MBKE3BElEB^B!|^MW3Wl 

BagWSSBSTSIISIBWggEnTIg^llgBnWSflgBIWSBlKtSF 

iHiilii^  fciiS]9  i£3l  iMSIiMi  S;1MSSI1  SSSd  tis 


ggai 


47  83 


WEggCTI 


[nKffaWgBB33aMnfflB333IBBniBtWgKEnB3!gWSgi»fc>^ 


]1H3|jn?HBcBn»EH 
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Appendix  I 

Air  Traffic  Workload  Input  Technique 
1.1  ATWIT:  Inferential  Statistics 


Table  I-l.  ATWIT:  MANOVA  Results 


MANOVA,  adjusted  alpha=0.0253 

Wilks' 

Lambda 

Rao  R 
Form  2 

■ 

df  1 

df2 

/?-level 

Load 

.191 

29.692 

0.809 

29.692 

2 

14 

.000 

Monitoring 

.674 

3.392 

0.326 

3.392 

2 

14 

.063 

Active 

.148 

40.359 

0.852 

40.359 

2 

14 

.000 

Interval  1 

.788 

1.888 

0.212 

1.888 

2 

14 

.188 

Interval  2 

.379 

11.452 

0.621 

11.452 

2 

14 

.001 

Interval  3 

.316 

15.126 

0.684 

15.126 

2 

14 

.000 

Interval  4 

.172 

33.801 

0.828 

33.801 

2 

14 

.000 

Interval  5 

.128 

47.730 

0.872 

47.730 

2 

14 

.000 

Interval  6 

.324 

14.595 

0.676 

14.595 

2 

14 

.000 

Involvement 

.198 

28.428 

0.802 

28.428 

2 

14 

.000 

Low  Load 

.442 

8.837 

0.558 

8.837 

2 

14 

.003 

High  Load 

.131 

46.296 

0.869 

46.296 

2 

14 

.000 

Interval  1 

.319 

14.929 

0.681 

14.929 

2 

m 

.000 

Interval  2 

.309 

15.651 

0.691 

15.651 

2 

14 

.000 

Interval  3 

.255 

20.435 

0.745 

20.435 

2 

14 

.000 

Interval  4 

.227 

23.841 

0.773 

23.841 

2 

14 

.000 

Interval  5 

.142 

42.264 

0.858 

42.264 

2 

14 

.000 

Interval  6 

.175 

32.917 

0.825 

32.917 

2 

14 

.000 

Time-on-Task 

.081 

6.807 

0.919 

6.807 

10 

6 

.015 

Load  X  Involvement 

.221 

24.655 

0.779 

24.655 

2 

14 

.000 

Load  X  Time-on-Task 

.086 

6.339 

0.914 

6.339 

10 

6 

.017 

Involvement  x  Time-on-Task 

.037 

15.518 

0.963 

15.518 

10 

6 

.002 

Load  X  Involvement  x  Time-on-Task 

.197 

2.445 

0.803 

2.445 

10 

6 

,143 

Table  1-2.  ATWIT  Rating:  ANOVA  Results 


ATWTT  Rating 

Means 

sqr 

Effect 

Means 

sqr 

Error 

F(dfl,2) 

1,15 

p-level 

Load 

232.815 

5.510 

42.257 

.000 

Monitoring 

21.333 

3.100 

6.882 

.019 

Active 

287.630 

3.864 

74.447 

.000 

Interval  1 

6.891 

1.757 

3.921 

.066 

Interval  2 

34.516 

1.416 

24.382 

.000 

Interval  3 

47.266 

1.699 

27.820 

.000 

Interval  4 

72.250 

2.317 

31.187 

.000 

Interval  5 

64.000 

1.533 

41.739 

.000 

Interval  6 

30.250 

1.350 

22.407 

.000 

Involvement 

563.086 

10.058 

55.983 

.000 

Low  Load 

112.547 

5.969 

18.855 

.001 

High  Load 

526.688 

5.543 

95.018 

.000 

Interval  1 

37.516 

1.182 

31.731 

.000 

Interval  2 

47.266 

1.566 

.000 

Interval  3 

102.516 

2.416 

42.439 

.000 

Interval  4 

105.063 

3.063 

34.306 

.000 

Interval  5 

132.250 

1.983 

66.681 

.000 

Interval  6 

175.563 

2.662 

65.939 

.000 

Time-on-Task 

5.446 

0.772 

7.056 

.000 

Load  X  Involvement 

76.148 

1.454 

52.372 

.000 

Load  X  Time-on-Task 

4.47  L 

0.912 

4.900 

.001 

Involvement  x  Time-on-Task 

7.417 

0.563 

13.180 

.000 

Load  X  Involvement  x  Time-on-Task 

2.942 

1.154 

2.549 

.035 

Table  1-3.  ATWIT  Latency:  ANOVA  Results 


ATWTT  Latency 

Means 

sqr 

Effect 

Means 

sqr 

Error 

F(dfl.2) 

1,15 

p-level 

Load 

2.344 

5.944 

0.394 

.539 

Involvement 

168.010 

25.555 

6.574 

.022 

Time-on-Task 

1.585 

7.832 

0.202 

.961 

Load  X  Involvement 

7.594 

13.483 

0.563 

.465 

Load  X  Time-on-Task 

19.519 

8.385 

2.328 

.051 

Involvement  x  Time-on-Task 

2.860 

7.258 

0.394 

.851 

Load  X  Involvement  x  Time-on-Task 

11.094 

9.516 

1.166 

.334 
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1.2.  ATWIT:  5-  Minute  Interval  Descriptive  Statistics  Based 


Table  1-4.  ATWIT  Rating  by  Load,  Involvement,  and  Time 


ATWIT 

Rating 

Low 

High 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

2.81 

1.38 

2.88 

1.59 

2.84 

1.46 

2.69 

1.45 

2.88 

1.41 

2.78 

1.41 

2.13 

1.15 

3.19 

1.87 

2.66 

1.62 

2.25 

1.13 

2.81 

1.83 

2.53 

1.52 

2.19 

3.44 

2.13 

2.81 

1.84 

wm 

1.98 

1.72 

_ 

tliJM 

Active  Control 

3.75 

■E! 

5.00 

2.07 

3.13 

1.31 

5.88 

2.39 

4.50 

2.36 

4.00 

1.55 

6.38 

1.67 

5.19 

1.99 

3.25 

2.32 

6.94 

1.81 

5.09 

2.77 

4.31 

2.12 

7.06 

1.65 

5.69 

2.33 

5.00 

2.13 

6.88 

2.00 

5.94 

2.24 

3.28 

1.37 

3.94 

2.11 

3.61 

1.80 

2.91 

1.38 

4.38 

2.46 

3.64 

2.11 

3.06 

1.64 

4.78 

2.38 

3.92 

2.21 

2.75 

1.87 

4.88 

2.76 

3.81 

2.57 

3.25 

2.03 

5.25 

2.63 

4.25 

2.54 

3.59 

2.26 

4.97 

2.75 

4.28 

2.59 

iii» 

mm 

ilii 

1-3 


Table  1-5.  ATWH  Latency  by  Load,  Involvement,  and  Time 


ATwrr 

Latency 

(seconds) 

Low 

ffigh 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

2.438 

1.094 

2.688 

1.250 

2.563 

1.162 

2.313 

1.014 

2.188 

1.167 

2.250 

1.078 

2.250 

1.125 

2.563 

1.315 

2.406 

1.214 

2.625 

1.544 

2.438 

1.548 

2.531 

1.524 

3.000 

2.033 

2.500 

1.366 

2.750 

1.723 

2.469 

2.188 

1.109 

2.438 

1.900 

ileUi 

Hi 

ilg 

iiM&i 

Active  Control 

2.563 

2.065 

4.750 

6.148 

3.656 

4.646 

2.625 

1.544 

5.500 

5.704 

4.063 

4.362 

4.250 

5.335 

2.875 

1.360 

3.563 

3.893 

4.438 

4.746 

2.688 

1.815 

3.563 

3.645 

4.625 

5.340 

4.125 

4.924 

4.375 

5.059 

liM 

2.750 

2.438 

1.605 

3.469 

4.544 

2.953 

3.420 

2.438 

1.343 

4.031 

4.337 

3.234 

3.284 

3.438 

3.951 

2.656 

1.450 

3.047 

2.978 

3.719 

3.665 

2.594 

1.583 

3.156 

2.858 

3.656 

4.209 

3.156 

3.647 

3.406 

3.915 

1 

iimi 

llBI 

Appendix  J 

Situation  Presence  Assessment  Method 

J.l.  Inferential  Statistics 

Table  J-1.  SPAM  Latency;  ANOVA  Results 


ANOVA 

df  Effect 

Means  sqr 
Effect 

Error 

Means  sqr 
Error 

f(dfl,2) 

p-level 

Load 

1 

82.140 

47 

26.822 

3.062 

.087 

Monitoring 

1 

53.130 

47 

22.895 

2.321 

.134 

Active 

1 

404.260 

47 

25.407 

15.912 

.000 

Involvement 

1 

1029.660 

47 

16.533 

62.278 

.000 

Low  Load 

1 

80.860 

47 

22.884 

3.533 

.066 

High  Load 

1 

1324.050 

47 

15.129 

87.520 

.000 

Type 

1 

0.220 

47 

27.139 

0.008 

.929 

Load  X  Involvement 

1 

375.250 

47 

21.480 

17.470 

.000 

Load  X  Type 

1 

20.167 

47 

31.945 

0.631 

.431 

Involvement  x  Type 

1 

25.627 

47 

31.500 

0.814 

.372 

Load  X  Involvement  x  Type 

1 

3.154 

47 

17380 

0.181 

.672 

Table  J-2.  SPAM  Query  Time:  ANOVA  Results 


ANOVA 

df  Effect 

Means  sqr 
Effect 

Error 

Means  sqr 
Error 

F(dfl,2) 

p-level 

Load 

1 

0.454 

47 

15.756 

0.029 

.866 

Involvement 

1 

0.023 

47 

13.802 

0.002 

.967 

Type 

1 

0.055 

47 

10.592 

0.005 

.943 

Load  X  Involvement 

1 

11.207 

47 

9.930 

1.129 

.294 

Load  X  Type 

1 

0.060 

47 

10.510 

0.006 

.940 

Involvement  x  Type 

1 

13.425 

47 

9.662 

1.389 

.244 

Load  X  Involvement  x  Type 

1 

22.234 

47 

10.896 

2.041 

.160 

Table  J-3.  SPAM  Response  Time:  ANOVA  Results 


ANOVA 

df  Effect 

Error 

F(dfl,2) 

Load 

1 

66.500 

47 

14.087 

4.721 

.035 

Involvement 

1 

42.135 

47 

7.444 

5.661 

.021 

Type 

1 

4.770 

47 

11.171 

0.427 

.517 

Load  X  Involvement 

1 

50.750 

47 

11.696 

4.339 

.043 

Load  X  Type 

1 

63.700 

47 

12.692 

5.019 

.030 

Involvement  x  Type 

1 

48.025 

47 

8.517 

5.639 

.022 

Load  X  Involvement  x  Type 

1 

119.038 

47 

9.340 

12.745 

.001 

Table  J-4.  SPAM  Response  Time  for  Present  Questions:  ANOVA  Results 


J-1 


Simple  Effects,  Present 
Questions 

df  Effect 

Means  sqr 
Effect 

Error 

Load 

1 

130.185 

47 

Monitoring 

1 

291.904 

47 

Active 

1 

0.901 

47 

Involvement 

1 

90.064 

47 

Low  Load 

1 

5.320 

47 

High  Load 

1 

247.363 

47 

Load  X  Involvement 

1 

162.619 

47 

Means  sqr  i^(dfl,2) 
Error 


13.950 


14.192 


8.218 


5.748 


4.994 


9.214 


9.332 


20.568 


0.110 


15.669 


1.065 


26.847 


19.222 


Table  J-5.  SPAM  Response  Time  for  Future  Questions:  ANOVA  Results 


df  Effect 

Means  sqr 
Effect 

Error 

Means  sqr 
Error 

F(dfl,2) 

1 

0.015 

47 

12.829 

0.001 

1 

0.096 

47 

10.213 

beh 

1 

7.169 

47 

12.576 

0.570 

Simple  Effects,  Future 
Questions 


Load 


Involvement  _ 


Load  X  Involvement 


J.2.  Scenario  Based  Descriptive  Statistics 

Table  J-6.  SPAM  Response  Time  by  Load,  Involvement,  and  Question  Type 


High  Load 


Low  Load 


Means 


Means 

SD 

Means 

6.21 

4.91 

4.48 

3.80 

3.83 

3.99 

2.86 

3.10 

2.72 

3.60 

4.03 

3.53 

4.31 

3.79 

3.52 

3.70 

4.01 

3.61 

MM 

Appendix  K 

Real  Time  Objective  Performance 

K.I.  Dependent  Variables 


Table  K-1 .  System  and  Performance  Measures 


Performance  Data 

Conflicts: 

No.  Conflicts 

- 

Dur.  Conflicts 

seconds 

Conflict  API 

-- 

No.  Longitudinal  conflicts 

Closest-point-of-approach  (feet) 

feet  (meters) 

Horizontal  separation  at  CPA  (feet) 

Vertical  separation  at  CPA  (feet) 

■HUH 

Average  System  Activity  CMAV 

Altitude  Changes 

- 

Heading  Changes 

- 

No.  Speed  changes 

“ 

Handoff  Efficiency: 

No.  Hand-offs  outside  boundary 

Communications: 

No.  Ground-to-air  contacts 

-- 

Dur.  Ground-to-air  contacts 

seconds 

No.  Pilot  message  key  strokes 

-- 

K.2.  Inferential  Statistics 


Table  K-2.  DRA  Altitude,  Heading,  and  Speed  Changes:  MANOVA  Results 


MANOVA,  adjusted  alpha=0.0169 

RaoR 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

df  1 

df2 

Load 

.244 

13.429 

0.756 

13.429 

3 

13 

.000| 

Table  K-3.  DRA  Altitude,  Heading,  and  Speed  Changes:  ANOVA  Results 


Effect  of  Task  Load 

Means  sqr  Effect 

Means  sqr  Error 

F(dfl,2) 

1,15 

> 

1 

Altitude  Changes 

2.183 

0.152 

14.352 

.002 

Heading  Changes 

0.197 

0.071 

2.763 

.117 

Speed  Changes 

0.078 

0.015 

5.058 

.040 

Table  K-4.  DRA  Distance  and  Time  Under  Control:  MANOVA  Results 


MANOVA,  adjusted  alpha=0.0253 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

df  1 

df2 

■ 

Load 

.074 

87.291 

0.926 

87.291 

2 

14 

.ooo| 

Table  K-5.  DRA  Distance  and  Time  Under  Control:  ANOVA  Results 


Effect  of  Task  Load 

Means  sqr  Effect 

Means  sqr  Error 

F(dfl,2) 

1,15 

/?-level 

Distance 

1587.033 

24.671 

64.328 

.000 

Time 

128552.000 

1615.184 

79.590 

.000 

Table  K-6.  DRA  PTT:  MANOVA  Results. 


MANOVA,  adjusted  alpha=0.0253 

Wilks' 

Lambda 

Rao  R 
Form  2 

Pillai- 

Bartlett 

Trace 

V 

dfl 

df2 

Load 

.658 

3.115 

0.342 

3.115 

2 

12 

.08 1| 

Table  K-7.  DRA  PTT:  ANOVA  Results 


Effect  of  Task  Load 

Means  sqr  Effect 

Means  sqr  Error 

F(dn,2) 

1,15 

/^-level 

Number 

0.560 

0.100 

5.597 

.034 

Duration 

7.349 

1.252 

5.870 

.031 

K.3.  Scenario  Based  Descriptive  Statistics 

Table  K-8.  Number  of  Altitude  Changes:  Mean  and  Standard  Deviations  by  Load. 


Number  of 
altitude  changes 
per  aircraft 

Mean 

SD 

Means 

SD 

Low 

1.39 

0.42 

High 

1.91 

0.45 

K-2 


Table  K-9.  Number  of  Heading  Changes:  Mean  and  Standard  Deviations  by  Load 


Number  of 
heading  changes 
per  aircraft 

Mean 

SD 

Means 

SD 

Low 

0.55 

0.27 

High 

0.23 

Table  K-10.  Number  of  Speed  Changes:  Mean  and  Standard  Deviations  by  Load 


Number  of 
speed  changes 
per  aircraft 

Mean 

SD 

Means 

SD 

Low 

0.22 

0.15 

High 

0.13 

0.13 

Appendix  L 

Subject  Matter  Expert  Rating  Form 
Table  L-1.  Providing  ATC  Information  by  Load  and  Involvement 


Providing  Essential 
Air  Traffic  Control 
Information 

Providing  Additional 
Air  Traffic  Control 
Information 

Providing 

Coordination 

Means 

SD 

Means 

SD 

Means 

SD 

Low  Load 

6.47 

0.80 

6.84 

0.68 

5.28 

0.81 

High  Load 

6.06 

1.05 

6.47 

1.08 

4.88 

1.34 

6.27 

0.95 

6.66 

0.91 

5.08 

1.12 

Table  L-2.  Prioritizing  by  Load 


Taking  Actions  in  an 
Appropriate  Order  of 
Importance 

Preplanning  Control 
Actions 

Handling  Control 
Tasks  for  Several 
Aircraft 

Marking  Right  Strips 
while  Performing 
Other  Tasks 

Means 

SD 

Means 

SD 

Means 

SD 

Means 

SD 

Low  Load 

6.72 

0.81 

6.34 

1.62 

6.75 

0.84 

6.03 

1.67 

High  Load 

6.00 

1.02 

4.72 

1.87 

6.00 

1.37 

5.22 

1.79 

6.36 

0.98 

5.53 

1.92 

6.38 

1.19 

5.63 

HKEB 

Table  L-3.  Attention  and  Situation  Awareness  by  Load 


Maintaining  Situation 
Awareness 

Ensuring  Positive 
Control 

Detecting  Pilot 
Deviations  from 
Control  Instructions 

Correcting  Errors  in  a 
Timely  Manner 

Means 

SD 

Means 

SD 

Means 

SD 

Means 

SD 

Low  Load 

5.34 

1.75 

6.47 

1.44 

6.81 

1.42 

6.84 

0.85 

High  Load 

4.00 

1.67 

5.84 

1.19 

6.28 

1.05 

6.09 

1.17 

4.67 

1.83 

6.16 

1.35 

6.55 

1.27 

6.47 

1.08 

Table  L-4.  Detecting  Pilot  Deviations  from  Control  Instructions  by  Load 


Detecting  Pilot 
Deviations  from 
Control  Instructions 

Correcting  Errors  in  a 
Timely  Manner 

Maint 

Separat 

Resolving 

Con] 

aining 
ion  and 
Potential 
flicts 

Sequencing  Arrival 
and  Departure 
Aircraft  Efficiently 

Means 

SD 

Means 

SD 

Means 

SD 

Means 

SD 

LowLoad 

6.81 

1.42 

6.84 

0.85 

6.56 

1.88 

5.91 

1.67 

High  Load 

6.28 

1.05 

6.09 

1.17 

5.41 

2.39 

5.81 

1.45 

6.55 

1.27 

6.47 

1.08 

5.98 

2.21 

5.86 

1.55 

L-1 


Table  Lr-5.  Using  Control  Instructions  Effectively  by  Load 


Using  Control 
Instructions 
Effectively 

Using  Proper 
Phraseology 

Communicating 
Clearly  and 
Efficientiy 

Listening  to  Pilot 
Readbacks  and 
Requests 

Means 

SD 

Means 

SD 

Means 

SD 

Means 

SD 

Low  Load 

6.38 

1.54 

5.81 

0.97 

6.88 

0.79 

7.16 

0.57 

High  Load 

1.48 

5.75 

1.08 

6.34 

1.15 

6.44 

1.22 

6.19 

1.51 

5.78 

1.02 

6.61 

1.02 

6.80 

1.01 

Table  L-6.  Showing  Knowledge  of  LOAs  and  SOPs  by  Load 


Showing  Knowledge 
of  LOAs  and  SOPs 

Showing  Knowledge 
of  Aircraft 
Capabilities  and 
Limitations 

Showing  Effective 
Use  of  Equipment 

Means 

SD 

Means 

SD 

Means 

SD 

Low  Load 

7.06 

0.80 

6.84 

0.99 

7.00 

0.72 

High  Load 

6.38 

1.41 

6.19 

1.18 

6.00 

1.34 

6.72 

1.19 

6.52 

1.13 

6.50 

1.18 

Table  L-7.  Showing  Effective  Use  of  Equipment  by  Load 


Showing  Effective  Use  of 
Equipment 

Means 

SD 

Low  Load 

7.00 

0.72 

High  Load 

mm 

1.34 

6.50 

1.18 

L-2 


Appendix  M 
Recall 


M.l.  Inferential  Statistics 


Table  M-1.  Percent  Correct  Recall:  ANOVA  Results 


df  Effect 

df  Error 

Means 

sqr 

Error 

F 

Load 

1 

4204.803 

12 

169.754 

24.770 

.000 

Involvement 

1 

1109.539 

12 

186.983 

5.934 

.031 

Load  X  Involvement 

1 

104.739 

12 

241.403 

0.434 

.523 

M.2.  Scenario  Based  Descriptive  Statistics 

Table  M-2.  Percent  Correctly  Placed  Data  Block  by  Load  and  Involvement 


Percent  Corret 

Low  Task  Load 

ErtSillfBliliill 

Means 

SD 

Monitoring 

51.99 

36.85 

Active  Control 

64.07 

43.25 

Appendix  N 

Post-Scenario  Questionnaire 


N.l.  Inferential  Statistics 
N.1.1  Realism 

Table  N-1.  Realism:  MANOVA  Results 


Wilks’ 

Lambda 

RaoR 
Form  2 

Pillai- 

Bartlett 

Trace 

■ 

df  1 

df2 

Load 

.319 

14.932 

0.681 

14.932 

2 

14 

.000 

Monitoring 

.837 

1.366 

0.163 

1.366 

2 

14 

.287 

Active 

.163 

36.009 

0.837 

36.009 

2 

14 

.000 

Involvement 

.233 

23.107 

0.767 

23.107 

2 

14 

.000 

Low  Load 

.338 

13.715 

0.662 

13.715 

2 

14 

.001 

High  Load 

.212 

25.988 

0.788 

25.988 

2 

14 

.000 

Load  X  Involvement 

.429 

9.300 

0.571 

9.300 

2 

14 

.003 

Table  N-2.  Realism:  ANOV A  Results 


Realism 

Means 

sqr 

Effect 

Means  sqr  Error 

F(dfl,2) 

1,15 

/?-level 

Load 

0.141 

0.741 

0.190 

.669 

Involvement 

54.391 

7.257 

7.495 

.015 

Load  X  Involvement 

0.016 

2.349 

0.007 

.936 

Table  N-3.  Representativeness:  ANOVA  Results 


Representativeness 

Means 

sqr 

Effect 

Means  sqr  Error 

F(dfl,2) 

1,15 

p-level 

Load 

0.016 

1.949 

0.008 

.930 

Involvement 

28.891 

5.357 

5.393 

.035 

Load  X  Involvement 

1.266 

1.866 

0.678 

.423 

N.1.2.  Difficulty 


Table  N-4.  Diffulty:  MANOVA  Results 


Wilks' 

Lambda 

Rao  R  Form  2 

Pillai-Bartlett 

Trace 

V 

dfl 

df2 

Load 

.319 

14.932 

0.681 

14.932 

2 

14 

.000 

.837 

1.366 

0.163 

1.366 

2 

14 

.287 

Active 

.163 

36.009 

0.837 

36.009 

2 

14 

.000 

Involvement 

.233 

23.107 

0.767 

23.107 

2 

14 

.000 

Low  Load 

.338 

13.715 

0.662 

13.715 

2 

14 

.001 

High  Load 

.212 

25.988 

0.788" 

25.988 

2 

14 

.000 

Load  X  Involvement 

.429 

9.300 

0.571 

9.300 

2 

14 

.003 

Table  N-5.  Working  Hard:  ANOVA  Results 


Hard 

Means 

sqr 

Effect 

Means  sqr  Error 

F(dfl,2) 

1,15 

/7-level 

Load 

39.063 

1.763 

22.163 

.000 

Monitoring 

3.781 

1.715 

2.205 

.158 

Active 

47.531 

1.198 

39.678 

.000 

Involvement 

175.563 

4.329 

40.553 

.000 

Low  Load 

47.531 

2.531 

18.778 

.001 

High  Load 

140.281 

2.948 

47.587 

.000 

Load  X  Involvement 

12.250 

1.150 

10.652 

.005 

Table  N-6.  Scenario  Difficulty:  ANOVA  Results 


Difficulty 

Means 

sqr 

Effect 

Means  sqr  Error 

F(dfl,2) 

1,15 

p-level 

Load 

33.063 

1.263 

26.188 

.000 

Monitoring 

3.125 

1.592 

1.963 

.182 

Active 

40.500 

1.700 

23.824 

.000 

Involvement 

72.250 

3.783 

19.097 

.001 

Low  Load 

13.781 

1.248 

11.043 

.005 

High  Load 

69.031 

4.565 

15.123 

.001 

Load  X  Involvement 

10.563 

2.029 

5.205 

.038 

N-2 


N.1.3.  Interference 


Table  N-7.  Interference:  MANOVA  Results 


MANOVA,  adjusted 
alpha=0.0253 

Wilks' 

Lambda 

Rao  R  Form  2 

(  4, 12) 

Pillai-Bartlett 

Trace 

V(4,12) 

dfl 

df2 

/?-level 

Load 

.467 

7.981 

0.533 

7.981 

2 

14 

.005 

Involvement 

.484 

7.475 

0.516 

7.475 

2 

14 

Load  X  Involvement 

.695 

3.065 

0.305 

3.065 

2 

14 

.079| 

Table  N-8.  ATWIT  Interference:  ANOVA  Results 


ATwrr 

Means 

sqr 

Effect 

Means  sqr  Error 

f(dfl,2) 

1,15 

1 

a. 

Load 

15.016 

0.882 

17.019 

.001 

Involvement 

19.141 

1.541 

12.424 

.003 

Load  X  Involvement 

5.641 

1.041 

5.420 

.034 

N.1.4.  Situation  Awareness 


Table  N-9.  Situation  Awareness:  MANOVA  Results 


MANOVA,  adjusted 
alpha=0.0127 

Wilks' 

Lambda 

Rao  R  Form  2 
(  4, 12) 

Pillai-Bartlett 

Trace 

V(4,12) 

dfl 

df2 

Load 

.340 

5.824 

0.660 

5.824 

4 

12 

.008 

Monitoring 

.531 

2.645 

0.469 

2.645 

4 

12 

.086 

Active 

.264 

8.372 

0.736 

8.372 

4 

12 

.002 

Involvement 

.416 

4.218 

0.584 

4.218 

4 

12 

.023 

Low  Load 

.509 

2.893 

0.491 

2.893 

4 

12 

.069 

High  Load 

.379 

4.923 

0.621 

4.923 

4 

12 

.014 

Load  X  Involvement 

.406 

4.384 

0.594 

4.384 

4 

12 

,021 

Table  N-10.  Overall  Situation  Awareness:  ANOVA  Results 


Overall  SA 

Means 

sqr 

Effect 

Means 
sqr  Error 

F(dfl,2) 

1,15 

p-level 

Load 

1.000 

1.667 

0.600 

.451 

Involvement 

1.563 

3.429 

0.456 

.510 

Load  X  Involvement 

12.250 

1.917 

6.391 

.023 

N-3 


Table  N-1 1.  Situation  Awareness  for  Current  Locations:  ANOVA  Results 


SA  for  current 
locations 

Means 

sqr 

Effect 

F(dfl,2) 

1,15 

/7-level 

Load 

18.063 

0.796 

22.696 

.000 

Involvement 

4.000 

3.933 

1.017 

.329 

Load  X  Involvement 

10.563 

1.763 

5.993 

.027 

Table  N-12.  Situation  Awareness  for  Projected  Locations:  ANOVA  Results 


SA  for  projected 
locations 

Means 

sqr 

Effect 

F(dfl,2) 

1,15 

> 

1 

Load 

13.141 

1.507 

8.718 

.010 

Involvement 

2.641 

4.807 

0.549 

.470 

Load  X  Involvement 

0.141 

1.907 

0.074 

.790 

Table  N-1 3.  Situation  Awareness  for  Potential  Violations:  ANOVA  Results 


SA  for  potential 
violations 

F(dfl,2) 

1,15 

p-level 

Load 

30.250 

2.283 

13.248 

.002 

Involvement 

2.250 

3.550 

0.634 

.438 

Load  X  Involvement 

1.000 

2.033 

0.492 

.494 

Table  N-14.  Quality  of  Control:  ANOVA  Results 


Load 

Means 

sqr 

Effect 

Means  sqr 
Error 

F(dfl,2) 

1,15 

> 

1 

Quality  of  Control 

18.000 

1.333 

13.500 

.002| 

N  0.  .Srp-nario  Based  Descriptive  Statistic 
N.  2.1  Realism 

Table  N-1 5.  Realism:  Mean  and  SDs  by  Load  and  Involvement 


Realism 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD  1 

Monitoring 

mm 

5.3 

3.1 

mm 

mEB\ 

Active 

I^Q] 

1.2 

■HQ 

1.4 

1.3 

6.3 

6.3 

2.5 

6.3 

2.6 
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Table  N-16.  Representativeness:  Mean  and  SDs  by  Load  and  Involvement 


Representativeness 

Low 

Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

5.69 

2.98 

5.44 

2.63 

5.56 

2.77 

Active 

6.75 

1.57 

7.06 

2.11 

6.91 

1.84 

6.22 

2.41 

6.25 

2.49 

6.23 

2.43 

N.  2.2  Difficulty 


Table  N-17.  Working  Hard  by  Load  and  Involvement 


Working  Hard? 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

2.94 

1.95 

3.63 

2.42 

3.28 

2.19 

Active 

5.38 

1.67 

7.81 

1.72 

6.59 

2.08 

4.16 

2.17 

5.72 

2.96 

4.94 

2.70 

Table  N-18.  Difficulty  by  Load  and  Involvement 


Difficulty 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

4.00 

1.86 

4.63 

2.33 

4.31 

2.10 

Active 

5.31 

1.54 

7.56 

1.90 

6.44 

2.05 

4.66 

1.81 

6.09 

2.57 

5.38 

2.32 

N.  2.3  Interference 


Table  N-19.  ATWIT  Interference  by  Load  and  Involvement 


ATWIT  Interference 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

1.25 

0.58 

1.63 

0.89 

1.44 

0.76 

Active 

1.75 

0.77 

3.31 

2.09 

2.53 

1.74 

1.50 

0.72 

2.47 

1.80 

1.98 

1.44 

Table  N-20.  Oculometer  Interference  by  Load  and  Involvement 


Oculometer  Interference 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

2.31 

2.36 

2.69 

2.33 

2.50 

2.31 

Active 

2.56 

1.63 

3.06 

2.26 

2.81 

1.96 

2.44 

2.00 

2.88 

2.27 

2.66 

2.13 
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N.  2.4  Situation  Awareness 


Table  N-21.  Overall  Situation  Awareness  by  Load  and  Involvement 


Overall  SA 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

6.75 

2.08 

7.38 

1.45 

7.06 

1.79 

Active 

7.31 

1.35 

6.19 

1.22 

6.75 

1.39 

7.03 

1.75 

6.78 

1.45 

6.91 

1.60 

Table  N-22.  Situation  Awareness  for  Current  Aircraft  Position  by  Load  and  Involvement 


SA  for  Current  Aircraft 
Position 

Low  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

5.63 

2.39 

5.38 

2.28 

5.50 

2.30 

Active 

6.94 

2.35 

5.06 

1.69 

6.00 

2.23 

6.28 

2.43 

5.22 

1.98 

5.75 

2.26 

Table  N-23,  Situation  Awareness  for  Projected  Aircraft  Position  by  Load  and  Involvement 


SA  for  Projected  Aircraft 
Position 

Low  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

6.56 

2.22 

5.75 

2.02 

6.16 

2.13 

Active 

7.06 

2.17 

6.06 

1.77 

6.56 

2.02 

6.81 

2.18 

5.91 

1.87 

6.36 

2.07 

Table  N-24.  Situation  Awareness  for  Potential  Violations  by  Load  and  Involvement 


SA  for  Projected 

Violations 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

8.06 

1.39 

6.94 

2.32 

7.50 

1.97 

Active 

7.94 

1.34 

6.31 

1.92 

7.13 

1.83 

8.00 

1.34 

6.63 

2.12 

7.31 

1.89 

Table  N-25.  Quality  of  Control  by  Load  and  Involvement 


SA  for  Projected 

Violations 

Low  Load 

High  Load 

Means 

SD 

Means 

SD 

Means 

SD 

Monitoring 

8.06 

1.39 

6.94 

2.32 

7.50 

1.97 

Active 

7.94 

1.34 

6.31 

1.92 

7.13 

1.83 

8.00 

1.34 

6.63 

2.12 

7.31 

1.89 

N-6 


Appendix  O 
Coordination  Events 

Scenario  1;  Active  High 

Coordination  Events 

17:30 

•  Genera  High,  Bravo  High 

I  need  United  422  (BC  2024)  at  flight  level  330 

•  (give  initials) 

33:00 

•  Genera  High,  Charlie  High 

I  need  Carnival  1 1  (BC  0674)  at  flight  level  240 

•  (give  initials) 

36:30 

•  Genera  High,  Bravo  High 

I  need  Spirit  Wings  2249  (BC  4655)  at  250  knots. 

•  (give  initials) 

Notes: 


Scenario  2:  Monitoring  Low 


Coordination  Events 

19:30 

•  Genera  High  this  is  Genera  radio  with  a  NOTAM.  Advise  when  ready  to  copy 

•  Southeast  VOR  is  NOTAMed  out  of  service  until  further  advised. 

•  (give  initials) 

36:00 

•  Genera  High  this  is  the  Military  desk. 

Whiskey  500  is  active  now  surface  to  flight  level  430. 

•  (give  initials) 

45:00 

•  Genera  High  this  is  the  Military  desk. 

Whiskey  500  is  deactivated. 

•  (give  initials) 

Notes: 

Scenario  3:  Practice 


Coordination  Events 

18:30 

•  Genera  High,  Charlie  Center 

I  need  US  Air  891  (BC  2045)  at  flight  level  3 10 

•  (give  initials) 

29:00 

•  Genera  High,  Alpha  High 

I  need  Delta  957  (BC  2016)  at  flight  level  330 

•  (give  initials) 
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45:00  {as  soon  as  DAL259  is  flashed  to  controller  and  AMX656  has  switched,  don ’t  call  if 
AMX656  is  outside  of  boundary) 

•  Genera  High,  Alpha  High 

Aero  Mexico  656  (BC  0666)  is  requesting  lower,  request  control  reference  Delta  259  {BC  3742). 

•  (give  initials) 

•  Don’t  descend  AMX656 

46:00  (oj  soon  as  datablock  is  flashed  to  controller) 

•  Genera  High,  Bravo  High 

Reference  Air  Shuttle  471  {BC  2555),  I  incorrectly  entered  an  assigned  altitude  of  260  in  data 
block,  he  wants  flight  level  240. 

•  (give  initials) 

Notes: 

Scenario  4:  Active  Low 


Coordination  Events 

19:30 

•  Genera  High,  Alpha  High. 

Request  control  for  US  Air  2174  {BC  4611 ),  I  need  him  at  flight  level  310. 

•  (give  initials) 

35:30 

•  Genera  High,  Charlie  Center. 

Kiwi  421  {BC  3762)  is  looking  for  lower,  my  control  reference  US  Air  1273  {BC  2565). 

•  (give  initials) 

•  Call  typist,  descend  K1A421  to  flight  level  330 

43:00 

•  Genera  High,  Bravo  High. 

Request  US  Air  8303  {BC  4243)  and  Critter  505  {BC  0636)  cross  lower  at  250  knots. 

•  (give  initials) 

Notes: 

Scenarios:  Practice 

Coordination  Events 

26:00  {After  COA131  has  switched  frequency) 

•  Genera  High,  Bravo  High 

Request  control  for  lower  on  Continental  131  {BC  4232) 

•  (give  initials) 

•  Call  typist  and  descend  COA131  to  flight  level  290 

34:00  {If  SJI707  has  switched  frequency,  request  control  for  higher) 

•  Genera  High,  Bravo  High 

I  need  Sun  Jet  707  {BC  2033)  at  flight  level  330 

•  (give  initials) 

•  If  requested  control  for  higher  and  it  was  granted,  call  typist  and  climb  SJI707  to  flight  level 
330 

41:00  {as  soon  as  datablock  flashed  to  controller) 

•  Genera  High,  Alpha  High 

Northwest  1277  {BC  2023)  is  requesting  flight  level  330 
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•  (give  initials) 

•  If  controller  asks  “my  control  for  higher?”,  say  “approved” 

•  If  controller  tells  you  to  climb  the  aircraft,  call  typist  and  climb  NWA1277  to  flight  level  330 
48:00  {after  DAL609  has  switched  frequency) 

•  Genera  High,  Charlie  Center 

Request  control  for  lower  on  Delta  609  {BC  3733). 

»  •  (give  initials) 

•  If  controller  says  “approved”,  call  typist  and  descend  DAlb09  to  flight  level  290 
Notes: 

Scenario  6;  Practice 

Coordination  Events 

19:00  {as  soon  as  USAI647  is  flashed  to  controller) 

•  Genera  High,  Alpha  High 

US  Air  1647  {BC  4654)  and  Delta  83  {BC  2536)  both  have  assigned  speeds  of  240  knots 
indicated 

•  (give  initials) 

28:30  {after  USA242  has  switched  frequency) 

•  Genera  High,  Charlie  Center 

US  Air  242  {BC  3771 )  is  requesting  flight  level  350,  my  control  for  descent? 

•  (give  initials) 

•  If  controller  says  “approved”,  call  typist  and  descend  USA242  to  flight  level  350 
41:00  {as  soon  as  COAI228  is  flashed  to  controller) 

•  Genera  High,  Alpha  High 

Continental  1228  {BC  2056)  is  requesting  flight  level  270. 

•  (give  initials) 

54:00  {as  soon  as  USAI680  is  flashed  to  controller) 

•  Genera  High,  Alpha  High 

US  Air  1680  {BC  2067)  and  US  Air  656  {BC  2555)  both  have  assigned  speeds  of  235  knots 
indicated 

•  (give  initials) 

Notes: 

Scenario?:  Practice 

Coordination  Events 

22:00 

•  Genera  High,  Charlie  Center 

I  need  Delta  1041  {BC  0662)  at  flight  level  310 

•  (give  initials) 

31:00 

•  •  Genera  High,  Charlie  Center 

I  need  US  Air  1269  {BC  2527)  at  flight  level  310 
•5  •  (give  initials) 

42:00  {as  soon  as  UAS609  flashes  to  controller) 

•  Genera  High,  Alpha  High 

US  Air  609  {BC  2534)  is  requesting  flight  level  290 

•  (give  initials) 
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•  if  controller  says  “approved”  call  typist  and  descend  USA609  to  flight  level  290 

46:15 

•  Genera  High,  Alpha  High 

US  Air  1432  (BC  0617)  is  requesting  flight  level  370,  my  control  reference  Aero  Mexico  417 
(BC2565) 

•  (give  initials) 

•  if  controller  says  “approved”  call  typist  and  climb  USA1432  to  flight  level  370 
Notes 

Scenarios:  Monitoring  High 

Coordination  Events 

26:00 

•  Genera  High  this  is  Genera  radio  with  a  NOTAM.  Advise  when  ready  to  copy. 

•  Northeast  VOR  is  NOTAMed  out  of  service  until  further  advised. 

•  (give  initials) 

36:00 

•  Genera  High  this  is  Genera  radio  with  a  NOTAM.  Advise  when  ready  to  copy. 

•  Runway  1 8  left  36  right  at  Uptown,  NOTAM  closed  for  mowing. 

•  (give  initials) 

42:00 

•  Genera  High,  Genera  Radio 

There  is  a  forest  fire  reported  about  30  miles  south  of  the  Center  VOR,  have  any  pilots  reported 
it? 

•  (give  initials) 

Notes: 
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Appendix  P 

Situation  Presence  Assessment  Method  Queries 

15:30 

Will  US  Air  1650  and  Continental  707  be  in  conflict  if  no  further  action  is 
taken,  yes  or  no? 

Yes 

No 

21:30 

Will  Lifeguard  99  Sierra  Fox  and  American  966  be  in  conflict  if  no  further 
action  is  taken,  yes  or  no? 

Yes 

No 

25:00 

Which  will  reach  the  Center  VOR  first,  Aeromexico  758  or  Carnival  11? 
Aeromexico  758 
Carnival  11 
28:00 

Are  there  any  speed  conflicts  on  the  J74  airway,  yes  or  no? 

Yes 

No 

32:00 

Which  is  traveling  at  a  faster  groundspeed,  US  Air  992  or  Spirit  Wings  2249? 
US  Air  992 
Spirit  Wings  2249 
41:00 

Which  is  at  a  higher  altitude,  US  Air  153  or  Delta  1676? 

US  Air  153 
Delta  1676 
15:30 

Will  Continental  707  and  US  Air  1650  be  in  conflict  if  no  further  action  is 
taken,  yes  or  no? 

Yes 

No 

21:30 

Will  American  966  and  Lifeguard  99  Sierra  Fox  be  in  conflict  if  no  further 
action  is  taken,  yes  or  no? 

Yes 

No 

25:00 

Which  will  reach  the  Center  VOR  first.  Carnival  11  or  Aeromexico  758? 
Aeromexico  758 
Carnival  11 
28:00 

Are  there  any  speed  conflicts  on  the  J74  airway,  yes  or  no? 

Yes 

No 

32:00 

Which  is  traveling  at  a  faster  groundspeed,  US  Air  992  or  Spirit  Wings  2249? 
US  Air  992 
Spirit  Wings  2249 
41:00 

Which  is  at  a  higher  altitude.  Delta  1676  or  US  Air  153? 

US  Air  153 
Delta  1676 
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21:00  , 

Which  will  leave  the  airspace  first.  Delta  1481  or  US  Air  2934? 

Delta  1481 
US  Air  2934 
26:30 

Which  is  traveling  at  a  faster  groundspeed,  Delta  1190  or  Jet  Ex  918? 

Delta  1190 
Jet  Ex  918 
32:30 

Which  has  a  higher  altitude,  Aeromexico  470  or  November  305  Alpha  Bravo? 

Aeromexico  470 

November  305  Alpha  Bravo 

39:00 

Which  will  reach  the  Center  VOR  first,  November  4  Mike  Delta  or  US  Air  145? 
November  4  Mike  Delta 
US  Air  145 
43:00 

Which  is  traveling  at  a  slower  groundspeed,  US  Air  124  or  Continental  1962? 
US  Air  124 
Continental  1962 
46:30 

Which  will  reach  the  Center  VOR  first,  US  Air  41  or  November  65  Romeo 
Charlie? 

US  Air  41 

November  65  Romeo  Charlie 
21:00 

Which  will  leave  the  airspace  first,  US  Air  2934  or  Delta  1481? 

Delta  1481 
US  Air  2934 
26:30 

Which  is  traveling  at  a  faster  groundspeed.  Jet  Ex  918  or  Delta  1190? 

Delta  1190 
Jet  Ex  918 
32:30 

Which  has  a  higher  altitude,  November  305  Alpha  Bravo  or  Aeromexico  470? 

Aeromexico  470 

November  305  Alpha  Bravo 

39:00 

Which  will  reach  the  Center  VOR  first,  US  Air  145  or  November  4  Mike  Delta? 
November  4  Mike  Delta 
US  Air  145 
43:00 

Which  is  traveling  at  a  slower  groundspeed.  Continental  1962  or  US  Air  124? 
US  Air  124 
Continental  1962 
46:30 

Which  will  reach  the  Center  VOR  first,  November  65  Romeo  Charlie  or  US  Air 
41? 

US  Air  41 

November  65  Romeo  Charlie 
24:00 

Which  will  reach  the  Center  VOR  first,  US  Air  1273  or  Delta  417? 

Delta  417 
US  Air  1273 
30:00 

Which  is  traveling  at  a  faster  groundspeed,  American  246  or  Delta  1033? 
Delta  1033 


P-2 


American  246 
34:00 

Which  is  traveling  at  a  slower  groundspeed,  US  Air  4095  or  Kacki  Blue  29? 
Kacki  Blue  29 
US  Air  4095 
41:00 

Which  will  reach  their  final  altitude  first.  Delta  1586  or  Trans  World  1432? 
Trans  World  1432 
Delta  1586 
44:00 

Which  will  reach  the  MIDLE  intersection  first.  Carnival  609  or  Critter  1176? 
Critter  1176 
Carnival  609 
47:00 

Which  has  a  higher  altitude.  Air  Jamaica  656  or  Continental  225? 

Continental  225 
Air  Jamaica  656 
19:20 

Which  has  a  higher  altitude,  Delta  1165  or  US  Air  2174? 

Delta  1165 
US  Air  2174 
23:15 

Which  will  reach  the  Center  VOR  first.  Critter  2250  or  Aeromexico  454? 
Critter  2250 
Aeromexico  454 
28:45 

Which  is  traveling  at  a  faster  groundspeed.  Continental  670  or  Carnival  471? 
Continental  670 
Carnival  471 
32:00 

Will  Lifeguard  1640  and  Delta  1165  be  in  conflict  if  no  further  action  is 
taken,  yes  or  no? 

Yes 

No 

37:00 

Which  should  reach  their  final  altitude  first,  US  Air  189  or  Continental  670 
US  Air  189 
Continental  670 
41:15 

Which  has  the  lower  altitude,  US  Air  1723  or  Critter  1658? 

US  Air  1723 
Critter  1658 
19:20 

Which  has  a  higher  altitude,  US  Air  2174  or  Delta  1165? 

Delta  1165 
US  Air  2174 
23:15 

Which  will  reach  the  Center  VOR  first,  Aeromexico  454  or  Critter  2250? 
Critter  2250 
Aeromexico  454 
28:45 

Which  is  traveling  at  a  faster  groundspeed.  Carnival  471  or  Continental  670? 
Continental  670 
Carnival  471 
32:00 

Will  Delta  1165  and  Lifeguard  1640  be  in  conflict  if  no  further  action  is 
taken,  yes  or  no? 
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Yes 

No 

37:00 

Which  should  reach  their  final  altitude  first,  Continental  670  or  US  Air  189? 
US  Air  189 
Continental  670 
41:15 

Which  has  the  lower  altitude.  Critter  1658  or  US  Air  1723? 

US  Air  1723 
Critter  1658 
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